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ABSTRACT 


This thesis discusses risk in Department of Defense 
(DoD) weapon systems acquisition. It uses the Marine 


Corps’ Advanced Amphibious Assault Vehicle (AAAV) as a case 





study in risk management strategy and techniques. 





The AAAV will provide the Marine Corps with a fast 





deploying, over-the-horizon, and waterborne insertion 
capability. The AAAV’s improvements over the currently 
fielded Amphibious Assault Vehicle (AAV) will provide 





Marines with a highly survivable and lethal weapon system 


ashore. 


Risk is the possibility of damage, injury or loss. 
The severity of a risk is determined by a combination of 
both the probability of an unfavorable event occurring and 


the severity of the event’s occurrence. 


Risks are present in virtually all DoD developmental 








programs. Programs suffer from risks in technical 





challenges, unstable system requirements, missing schedule 


milestones, unpredictable funding and cost overruns. 


The DoD currently uses techniques to mitigate risks 


inherent in advanced system development. This thesis 








analyzes the AAAV’s Program Definition and Risk Reduction 
(PDRR) acquisition phase risk management strategy. The 


thesis concludes by drawing from the lessons learned in the 





AAAV program during PDRR and analyzing the application of 








the lessons learned during the AAAV’s current acquisition 





phase, System Development and Demonstration (SDD). 
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I. INTRODUCTION 


A. PURPOSE 





This thesis xamines th Department of Defense (DoD) 
system acquisition risk management environment by analyzing 


the Marine Corps’ Advanced Amphibious Assault Vehicle 








(AAAV) program. To conduct this analysis, this thesis will 


discuss risk in the context of DoD development and 





procurement, current risk management practices in DoD and 


in the defense industry, and introduce the AAAV system to 





briefly familiarize the reader with the program. The 


analysis will concentrate on the AAAV Program Definition 





and Risk Reduction (PDRR) Acquisition phase. This thesis 





will discuss the AAAV’s System Development and 
Demonstration (SDD), the current Acquisition Phase, risk 
management strategy with respect to lessons learned during 


PDRR. This thesis will conclude by examining the AAAV’s 





SDD risk management practices and providing recommendations 
for managing risk in developmental weapons system 
acquisition based on the AAAV’s experiences. 
B. BACKGROUND 

Risk is the possibility of injury, damage or loss. In 
DoD systems acquisition, risks are the “chances of not 


achieving the results as planned.” (Forsberg, Mooz and 





Cotterman, 2000, p. 188) In weapon system development and 
procurement, planned results are meeting operational 
deficiencies throughout DoD on time, on budget and to a 
satisfactory performance level. The failure to satisfy the 


war fighter’s requirements can result in decreased 





effectiveness of the United States DoD. 


Risk is the “probability or likelihood of failing to 
achieve a particular outcome” and “the consequence or 
impact of failing to achieve that outcome.” (Defense 


Acquisition University (DAU), Risk Management Guide for DoD 





Acquisition, 2001, p. 5) A level of risk is determined by 
combining both the probability of the undesirable event 


occurring and the impact, or severity, of the event. There 
are many categories of risk. This thesis discusses 
technical risk, requirements risk, schedule risk and 


cost/funding risk. 


The DoD acquisition regulations are undergoing change 


at the time this thesis is being written. The AAAV program 





executed its risk management strategy based on then current 





DoD acquisition guidelines and regulations. This thesis 
discusses risk management practices designed to reduce, 
eliminate, transfer and accept risk in developmental 
programs. The purpose of this research and analysis is to 
present the risk management techniques the AAAV program has 
benefited from most. Additionally, this thesis will 


discuss which aspects of the program’s PDRR risk management 





strategy have led to the adoption of different techniques 

in SDD and discuss. why. The overall benefit of this 

research is to familiarize the reader with successful risk 

management practices in DoD acquisition. 

Cc. RESEARCH QUESTIONS 
The primary research question this thesis addresses 


is: 





e How have the lessons learned from the AAAV’s 
Program Definition and Risk Reduction (PDRR) Risk 
Management Strategy impacted the Program’s Risk 
Management Process during System Development and 
Demonstration (SDD)? 








2 


In order to answer the primary research question, this 
thesis will answer the following subsidiary questions to 


provide the necessary background information: 


e What are risk and risk management in Department 
of Defense (DoD) systems acquisition? 


e What techniques can DoD use to manage risk in 
developmental systems? 


e What is the AAAV program? 


e What are the lessons learned from the AAAV PDRR 
Risk Management Strategy? 





e What risk management approaches has the AAAV 
Program Office adopted to manage technical and 
programmatic risk during SDD? 


e What conclusions and recommendations can be drawn 
from this analysis? 


D. RESEARCH METHODOLOGY 


This author’s research methodology included extensiv 














literary and Internet searches. The primary forms of 
literature used wer DoD publications and guidelines, 
magazine articles and textbooks. The Internet provided a 





great deal of information on DoD risk management techniques 
and on the AAAV. Of greatest benefit to the research was 
the opportunity to visit the AAAV program office in 
Virginia. This author was able to interview Government 
program office as well as Prime Contractor personnel. The 
information and insights were invaluable to this effort. 
E. ORGANIZATION OF THE STUDY 

This thesis is organized into six chapters. A brief 


description of the chapters’ content follows. 


Chapter I introduces the thesis and the primary and 


subsidiary thesis questions. The purpose of this chapter 


is to provide a snapshot of the thesis and its intended 


benefit to readers. 


Chapter II provides background information on the DoD 


risk management environment. The chapter offers the reader 





the information necessary to better appreciate subsequent 
chapters. Chapter II discusses types of risk commonly 
encountered in defense acquisitions and presents risk 
management techniques used in weapon system procurement and 


development. 


Chapter III provides the reader with background 


information on the AAAV system and its acquisition history 





to date. The purpose of Chapter III is to familiarize the 
reader with the challenges and complexities of developing a 


system like the AAAV. 


Chapter IV discusses the AAAV PDRR risk management 








techniques. This chapter presents the data to be analyzed. 
The chapter will focus on five areas of risk management in 
the AAAV program during PDRR: 

e Information Technology Tools 

° Risk Management Process 

° Managing Risk Through the Contracting Process 

° Government and Prime Contractor Co-location 

e Test and Evaluation 


Chapter V analyzes the AAAV PDRR risk management 
strategy and introduces elements of the AAAV SDD _ risk 


management plan based on PDRR lessons learned. 





Chapter VI concludes the thesis by summarizing how the 
lessons learned from the AAAV’s PDRR- risk management 


strategy have helped shape the program’s current risk 


management practices in SDD. The thesis closes’ by 
presenting recommendations for managing risk in DoD 
acquisition programs and offering areas for further 
research and study in DoD acquisition risk management. 
F. SUMMARY 

The purpose of this chapter is to provide the reader 
with an overview of this thesis. The benefit of this 


research and analysis is to highlight successful risk 








management techniques in complex, developmental weapon 
systems. The techniques and procedures may have 
application to managing risk in any program or 
organization. 


The next chapter provides background information on 


the DoD risk management environment. 
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II. BACKGROUND 


A. INTRODUCTION 


This chapter defines risk in the context of Department 





of Defense (DoD) program management and systems 
acquisition. The chapter then analyzes the risk management 
and risk mitigation processes in DoD. It addresses the 


importance of striking a balance between risk acceptance 


and risk mitigation in a developmental weapon system 





program. This chapter concludes by exploring different 
risk management techniques commonly used throughout the DoD 
acquisition environment. 
B. RISK IN THE DEPARTMENT OF DEFENSE 

Risk is the possibility of injury, damage or loss. In 
program management, risks are the “chances of not achieving 
the results as planned.” (Forsberg, Mooz and Cotterman, 


2000, p. 188) With rapid technological growth and 





emerging, complex mission needs, risk exists in virtually 
all of today’s DoD developmental weapons systems. In 
Defense Acquisitions, loss refers to the impact of the risk 
to a program, which could be in the form of diminished 


performance, increased costs or schedule delays. Risk is 





the “probability or likelihood of failing to achieve a 
particular outcome” and “the consequence or impact of 


failing to achieve that outcome.” (Defense Acquisition 





University (DAU), Risk Management Guide for DoD 
Acquisition, 2001, p. 5) 


Risk, whether programmatic, technical, managerial, 





etc., is present in DoD developmental systems. Numerous 


risk areas exist in the acquisition environment, each 
posing a threat to the success of a program. 

dc Types of Risk 

Risks are future events that may or may not occur. In 
DoD acquisitions, risks are future events that may 
adversely affect a program’s cost constraints, schedule or 


performance requirements. The types of risk are often 





interrelated and are not always obvious. 


Risks are in the Program Management Office (PMO) 
(program plans, etc.); in support provided by 
other Government agencies; in threat assessments; 
and in prime contractor processes, ngineering 
and manufacturing processes, and technology. 
(Risk Management Guide for DoD Acquisition, 2001, 
p. 6-7) 














A Program Manager (PM) is faced with a wide assortment 


of risk types in a program. Identifying risk in a program 





is a vital step in managing the potential, negative impacts 





of risk. Risk analysis is the “process of examining each 


identified risk area to refine the description of the risk, 








isolating the cause, and determining the effects.” 
(Guidelines for Successful Acquisition of Software —- 


Intensive Systems (GSAM), 2000, p. 6-18) Before risk 





analysis and mitigation can be discussed, several types of 


risks that programs often face must be analyzed. 





Sources of risk can be generally classified, but are 
not limited to, one of the following categories: technical 
risk, requirements risk, schedule risk and cost/funding 
risk. (Risk Management Guide for DoD Acquisition, 2001, p. 
7) 


a. Technical Risk 


Technical risk is the “degree to which the 





technology proposed for the program has been demonstrated 
as capable of meeting all of the program’s objectives.” 


(Risk Management Guide for DoD Acquisition, 2001, p. 8) 





Technical risk refers to the maturity level of technology 


utilized in the system being developed. The main concern 





with technical risk is that the system will fail to perform 





to expected standards because of immature or _ poorly 





integrated technology. In software development, a great 


technical risk lies in the difficulty in measuring 








developmental progress through the use of Technical 





Performance Measurements (TPM). TPMs are metrics that a PM 





may use to measure progress in a program. Many TPMs used 





in DoD lend themselves to physical measurements: weight, 





height, voltage, power, etc. Given modern systems’ 


reliance on software to achieve technical objectives, an 





inability to accurately monitor softwar development 








a 


progress by means of a concrete TPM will continue to pose a 
great technical risk to a developmental program. 


b. Requirements Risk 





The requirements generation process produces 
information for decision makers on the projected mission 
needs of the war fighter. A system evolves from the 
President’s National Security Strategy (NSS), DoD’s 
National Military Strategy (NMS), through several layers of 


analysis and refinement until the issuance of the Mission 





Needs Statement (MNS). The MNS defines, in broad, general 








terms a deficient operational capability based on threat 


assessments. 


Mission needs are defined in broad operational 


terms in a Mission Needs Statement (MNS) document. Based 





on the MNS, services conduct Analyses of Alternatives (AoA) 





to assess the potential for application of fielded, DoD 





systems to meet the emergent requirement. If no suitable 


alternative exists within DoD, an Operational Requirements 





Document (ORD) is issued which initiates the development of 


a new system. (Systems Engineering Fundamentals, 2001, p. 





45) Requirements definition is vital to establishing and 





adhering to a strict timeline or schedule for the program. 


The Requirements Generation Process is one of 


three elements in the DoD’s principal decision support 





system. The system results in “identifying and documenting 





war fighting needs based on current or future mission 
deficiencies or technological opportunities.” (Systems 
Engineering Fundamentals, 2001, jon 27) Figure 1 


illustrates the evolving Requirements Generation Process 








from the issuance of the ORD through system fielding. 
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: =F @ Chui 
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that is Protected by a 
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® Item Spec id 
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* ICBM - ares 
“ SKE Cruise Missile 
® Cruise Missile 
Manned Bomber ‘@ Tomahawk [acu 
® Attack Aircraft #ALCM 
Figure 1. Requirements Generation Process, from 


(Test and Evaluation Management Guide, 2001). 
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Ensuring that a system’s requirements can be 
identified and established early and accurately greatly 
reduces the risk of requirements creep. Figure 2 


illustrates how th Requirements Generation Process 





overlaps with Acquisition Management and the Planning, 
Programming and Budgeting System (PPBS) and is a crucial 


element to the system development process. 












Acquisition 
Management 
System 








Planning, 
Programming, 
& Budgeting 
System 





Requirements 
Generation 
system 











Figure 2. Three DOD Decision Support Systems, 
from (CJCSI, 2001). 


The risk associated with a requirement is linked 
to the variability of the requirement. “Creeping” or 


changing requirements can lead to schedule delays and can 





significantly impact a program. Requirements risk is the 











“sensitivity of the program to uncertainty in the system 
description and requirements.” (Risk Management Guide for 


DoD Acquisition, 2001, p. 7) 


The ORD is reviewed several times throughout the 








life of a program. Each review may alter original 
requirements, which can initiate time consuming and costly 


Engineering Change Proposals (ECP). Such changes can 





negatively impact a program’s cost and schedule. Figure 3 
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shows the interface between the system lifecycle and the 


requirements analysis process. 
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Figure 3. Current Requirements and Acquisition 
Interface, from (CJCSI, 2001). 


The current requirements and acquisition 


interface contains significantly fewer opportunities to 





impact a program based on creeping mission requirements 
than the previous acquisition process; however, changing 
requirements at any time introduces the risk of costly 


design changes. Design changes late in a program’s life 





can be technologically challenging and costly to the 


Government. 
Requirements risk may occur as a result of any of 
the following: 


° Operational requirements not properly established 
or vaguely stated for program phase 





° Requirements are not stable 

° Required operating environment not described 

° Requirements do not address logistics and 
suitability 

° Requirements are too constrictive-identify 


specific solutions that force high cost (GSAM, 
2000, p. 6-29) 


12 





Without adequate and stable requirements 
definition early in the life of the program, the Program 
Management Office (PMO) may be forced to make costly 
changes in the system. 

c. Schedule Risk 

Program Managers are evaluated on the cost, 
schedule, and performance of their program. (DoD 5000.2-R, 
2002, pp. 21, 24) Many acquisition programs are driven by 
time, or schedule, rather than by significant events or 
milestones in the program’s progress. Many factors can 


influence a program’s ability to adhere to a_ specific 


schedule. Schedule risk is the “adequacy of the time 
allocated for performing the defined tasks, e.g., 
developmental, production, etc. This factor includes the 





effects of programmatic schedule decisions, the inherent 





errors in the schedule estimating technique used, and 


external physical constraints.” (Risk Management Guide for 





DoD Acquisition, 2001, p. 8) 


Virtually every risk area can degrade a program’s 


ability to maintain a schedule. In the design of a system, 





reliance on immature technology or an unproven development 











process can cause a program’s schedule to. slip. ie 
logisticians are not involved in the early system 
development process, inadequate supportability late in 
development or after fielding can result in the necessity 





to make engineering changes causing delays in the system’s 





Initial Operational Capability (IOC). 


Pressure exists for a PM to establish an 


acquisition lifecycl schedule early in the system’s life 








and to maintain that schedule throughout. Development time 


15S, 


estimates are based on several factors including parallel, 





or like-system development and contractor estimates. 
Department of Defense policy concerning acquisition 


schedule is as follows: 


Schedule parameters shall minimally include (in 
Acquisition Program Baseline (APB)) dates for 
program initiation, major decision points, and 
the attainment of initial operating capability 
(IOC). The PM may propose, for Milestone 
Decision Authority (MDA) approval, other, 
specific, critical, system events, as necessary. 
(DoD 5000.2-R, 2002, p. 22) 














A program’s risk of experiencing a schedule delay 





is compounded, for example, by th development and 





integration of new technologies, changing requirements and 
budget constraints, among others. A program unable to 
comply with an approved schedule may risk cancellation. 

d. Cost/Funding Risk 

Without funding, a program has no _ life. A 


detailed, total ownership cost (TOC) estimate is required 





upon initiation of a program. DoD guidelines are specific 


in their direction concerning the establishment of detailed 








cost estimates: 


Cost parameters shall identify TOC (broken-out 
into direct costs: research, development, test, 
and evaluation costs, procurement costs, military 
construction costs, operating and support costs 











(to include environmental, safety, and 
occupational health compliance costs), and the 
costs of acquisition items procured with 


operations and maintenance funds, if applicable. 
Cost figures shall reflect realistic estimates of 
the total program, including a thorough 
assessment of risk. (DoD 5000.2-R, 2002, p. 23) 





A PM clearly needs to provide an accurate, sound 





estimate on the TOC of the program before system 
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development begins. The risk is the inability to 


accurately predict costs given uncertainty at the outset of 





a program. The cost risk area is whether a program has 


“the ability to achieve the program’s life cycle cost 





objectives. This includes the ffects of budget and 





affordability decisions and the effects of inherent errors 
in the cost estimating technique(s) used (given that the 
technical requirements were properly defined).” (Defense 


Acquisition University, 2001, p. 8) 





Technical, requirement, schedule and cost/funding 
risks are but a few examples of many risk areas prevalent 


in defense acquisitions. To summarize the impact of each 








risk area and the interrelatedness of each, the Government 
Accounting Office (GAO) wrote in a report regarding 


acquisition risk: 





Once in a product development environment, 
external pressures to keep the program moving 
(such as preserving cost and schedule estimates 
to secure budget approval) become dominant. Ifa 
program manager decided that an additional year 
was needed to reach the desired level of 











technical maturity during the risk 
reduction/concept demonstration phase, the 
planned start of the engineering and 


manufacturing development phase could be delayed. 
This delay could jeopardize funding for that 
phase, thus risking the funding support for the 
entire program. (United States General Accounting 
Office, 2000, p. 16) 





Programs exist because a need or requirement 





exists to better support or equip war fighters. All the 
numerous risks surrounding a defense acquisition program 
threaten the DoD’s ability to respond to a specific mission 


need or leverage emerging technologies and improve our 


LS 


current war fighting capabilities. Therefore, it is 





imperative that program managers actively manage and 
mitigate risks in programs. The next section discusses 
risk management techniques in DoD. 
Cc. RISK MANAGEMENT 

While risk is the probability of a future event 
occurring and the impact of that event, risk management is 
concerned with “the outcome of future events and how to 
deal with uncertainty.” (Risk Management Guide for DoD 
Acquisition, 2001, p. 1) Throughout DoD, risk management 


is recognized as a vital management tool that spans the 





entire acquisition lifecycle from concept exploration to 


operations and support. (GSAM, 2000, p- 6-4) If 
implemented early into a program’s management, risk 
management becomes a way of life. The key to successfully 








managing risk is planning and forward thinking. 


To support these efforts, assessments should be 
performed as early as possible in the life cycle 
to ensure that critical technical, schedule and 
cost risks are addressed with mitigation actions 
incorporated into program planning and _ budget 
projections. (Defense Systems Management College, 
2001, p. 2-3) 











The remainder of this section will discuss) risk 





management practices and techniques commonly used in DoD. 


This thesis will break down and analyze risk 


management in four parts: (1) Risk Planning, (2) Risk 





Assessment, (3) Risk Mitigation, and (4) Risk Tracking. 





(Defense Acquisition Deskbook (DAD), 2002) 
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Plan 
(What, when, 


Monitor \ () Assess 
(Identify and 


and Report 
(Know what’ analyze) 















Handle 
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risk) 





A Continuous Interlocked Process—Not an Event 


Figure 4. Risk Management Continuum, from (DAU, 
Systems Engineering Fundamentals, 2001). 


1. Risk Planning 


Risk planning is the process of “developing and 





documenting an organized, comprehensive and interactive 
strategy and methods for identifying and tracking risk 


areas, developing risk-mitigation plans, performing 





continuous risk assessments to determine how risks have 


changed or what new risk exists.” (GSAM, 2000, p. 6-11) 








Planning for adequate resources is vital to implementing a 





risk management plan throughout the entire lifecycle of the 








program. The DoD 5000.2-R mandates that PMs include risk 


management in the acquisition strategy. 


Risk planning is a continuous effort throughout the 








life of a program. Risk planning is not a single event. 





(Systems Engineering Fundamentals, 2001, p. 134) Init ial 


planning includes “establishing a strategy; establishing 








goals and objectives; planning assessment, handling and 





monitoring activities; identifying resources, tasks and 


ili 


responsibilities and establishing a method to document and 








disseminate information on a continuous basis.” (Systems 
Engineering Fundamentals, 2001, p. 135) An example Risk 
Management Plan (RMP) outline is depicted in Figure 5 


below: 


Introduction 
Program Summary 
Definitions 
Risk Management Strategy and Approach 
Organization 
Risk Management Process and Procedures 


Risk Planning 
Risk Assessment 
Risk Handling 
Risk Monitoring 
Risk Management Information System, Documentation and Reports 





Figure 5. Risk Management Plan Outline/Format, 
from (Systems Engineering Fundamentals, 2001). 





An important aspect of risk planning is the 
identification of shortfalls, whether technical expertise 


or resources. Identifying shortfalls allows a program 





office to identify risk areas that may require additional 





augmentation or tracking. The RMP should be fully 
integrated into the program Acquisition Strategy. Within 
the framework of the Integrated Product and Process 
Development (IPPD) concept, assigning a Risk Management 


Coordinator (RMC) to a program office provides the team a 





focal point for risk management who is responsible for 


implementing and supervising the risk management process. 


Once the RMP has’ addressed the risk management 
strategy and organization, the next step is to identify or 
assess program risks. The next section will discuss Risk 


Identification/Assessment. 
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2: Risk Identification/Assessment 


Risks can be viewed as opportunities. “Risk and 
opportunity go hand in hand. Success cannot be achieved 
without some degree of risk.” (Carnegie Mellon Software 





Engineering Institute (SEI), 1999, p. 3) Opportunities are 


defined as “chances for progress or advancement” or 








“chances for improving the value of the project results.” 
(Forsberg, Mooz, Cotterman, 2000, p. 188) Programs and 


PM’s risk failure when they fail to identify program risks. 


One of the biggest problems a project manager 
faces is motivating team members to identify 
risks. You want to make everyone risk conscious. 
However, there is often that hesitancy to surface 
risks, lest one be labeled a worrier or negative 
thinker. You can’t mitigate it (risk) if you 
don’t know Tees there so it’s better to 
anticipate a lot of problems, some of which won’t 























happen, than too few and miss the “project 

killers.” (Forsberg, Mooz, Cotterman, 2000, op. 

193) 

This section discusses risk identification and 
assessment. 


Risk identification begins by compiling the program’s 








risk events. Examining each Work Breakdown Structure (WBS) 


product and process element in terms of the sources or 





areas of risk most easily identifies risk events. (Systems 





Engineering Fundamentals, 2001, p. 11) 





A WBS is a “means of organizing system development 


activities by examining the physical and architectural 





qualities of a system.” (Systems Engineering Fundamentals, 
2001, p. 85) The WBS enables PM’s to identify potential 


risk areas in development and system integration. 
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Figure 6. Example Work Breakdown Structure, from 
(Systems Engineering Fundamentals, 2001). 


The WBS enables an entire system to be visualized 


through a “logical breakdown of product elements into work 





packages.” (Systems Engineering Fundamentals, 2001, p. 86) 
Once risk areas are identified, a PM needs to categorize 
and prioritize risks elements in the program. 

a. Risk Analysis Process 

Through the IPPD process, program planners, 
engineers, logisticians and other functional area 


representatives discuss and analyze identified risk areas. 





The analysis includes determining the likelihood that a 
risk area will occur and the impact of the occurrence. 
Many tools exist to assist in this process. A risk matrix 


is a helpful tool to assess risk areas in programs. 
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RISK RATING 


ASSESSMENT GUIDE 


| ee | What isthe Likelihood 
The Risk Wil Happen? 
‘Srmall/Unlikely 
Acceptable/Likelyy 
Large’Highly Lkey 
Signi ficant/Near Certain 


123 4 5 


Consequence 


Level TechnicalPerformance and/ Schedule and/ = |mpact on Other 
or or Teams 
Minimal or No Impact Minimal or No Impact Minimal or No Impact None 
Acceptable with Some Additional Resources Required; <5% Some Impact 
Reduction in Margin Able to Meet Need Dates 


Acceptable with Minor Slip in Key Milestone; Not 5-7% Moderate Impact 

Significant Reduction in Able to Meet Need Dates 

Margin 

Acceptable, No Major Slip in Key Milestone or Major Impact 

Remaining Margin Critical Path Impacted 

Unacceptable Can't Achieve Key Team or Major Unacceptable 
Program Milestone 





Figure 7. Example Risk Matrix, from (GSAM, 2000). 


The matrix enables a PMO to individually analyze 
risk areas, determine the likelihood of occurrence and 
assess the impact of the risk on the program’s cost, 


schedule or performance. 


Risk assessments are categorized green, amber or 





red in ascending criticality. A PMO may elect to pay 
greater attention to amber or red items throughout the risk 
mitigation process than low risk, or “green” risks. 
Critical, or red, risks may be deemed unacceptable to a 
program and generate engineering change proposals (ECP) or 
an aggressive mitigation strategy to reduce the risk to an 


acceptable level. 


The assessments for probability of risk 
occurrence are based on program office personnel 
experience, Similar or parallel system development, 
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modeling and simulation, like-component mean time to 


failure and technology maturity, among others. The risk 





impact assessment is similarly determined. The following 


provides examples of risk impact determination criteria: 











° Comparisons with similar systems, 

° Relevant lessons-—learned studies, 

° Experience, 

e Results from tests and prototype development, 

e Data from engineering or other models, 

° Specialist and expert judgments, 

° Analysis of plans and related documents, 

° Modeling and simulation, 

° Sensitivity of analysis of alternatives. (Risk 
Management Guide for DoD Acquisition, 2001, p. 
15) 


Once the risk area probabilities and impacts are 





determined, the risks may be prioritized and rated based on 


greatest probability of occurrence and impact to the 


program. 
b. Risk Rating and Prioritization 
Risk ratings are indications of potential impact 
of risks on a= program. Risks are often rated and 
categorized as High, Moderate or Low. Risk ratings and 


prioritization are considered an integral part of risk 
analysis. Prioritizing risks is the first step in 


developing a risk mitigation strategy, focusing efforts 





first on risks that carry the greatest potential impact on 
the program. Several tools exist to assist the PMO to make 


preliminary judgments regarding risk classification. 
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Classification Matrix, 


(Systems Engineering Fundamentals, 


2001). 


The documented, prioritization is called a risk 
Watch List. 


2001, py 17) 


(Risk Management Guide for DoD Acquisition, 


A prioritized watch lists allows the PMO to 





visualize risk areas and concentrate management and 


leadership efforts where they are most needed. 


Area/Source Location Risk Event Proba- Conse- Risk 
Process bility quence Rating 
WBS 3.1 Design nat Very Severe High 
completed on time likely 





Figure 9. 
Management Guide for DoD Acquisition, 





Example Risk Rating Matrix, from (Risk 


2001). 
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Based on identified risk areas and their 
probability of occurrence and impact to the program, the 


PMO can develop a mitigation strategy. 


Idontify and List All Risks 


Establish & Risk Definition Matrix 
ard Assign Risks to a Risk Area 


ve _ Estetyd Deter owed ety m Pyne eee 





’ 
‘ 
° 
. 
. 
. 
‘ 

' 

' 

1 
‘ 


Establish 2 Risk Priority List 

+ Prioritize risk based om matix 
+ Establish crtical “hagh risk” list 
+ Eetablich a moderate risk” nt 





Figure 10. Initial Risk Identification and 
Prioritization, from (Risk Management Guide for 
DoD Acquisition, 2001). 





3. Risk Mitigation 


Risk mitigation is the process that “identifies, 





evaluates, selects, and implements options in order to set 





risk at acceptabl levels given program constraints and 





objectives.” (GSAM, 2000, Pp. 6-19) Risk mitigation 
includes determining what should be done to manage a 
particular risk, how often it should be done and reported, 
who is responsible for handling it and what the cost impact 
of managing the risk is. PM’s must determine the possible 


“consequences of action or inaction as well as conducting a 





cost-benefit analysis of mitigation actions.” (GSAM, 2000, 


P.- 6-20) Risk mitigation actions should also be closely 








tied to metrics that measure the success, progress or 


failure of a particular mitigation action. 
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A program’s RMC has several options regarding risk 


handling. RMCs may assess risk mitigation proposals based 





on the following criteria: 





° Can the option be feasibly implemented and still 
meet the user’s needs? 








° What is th xpected ffectiveness of the 
handling option in reducing program risk to an 
acceptable level? 





e Is the option affordable; based on both fiscal 
and time constraints? 





° What effect, if any, does the option have on the 
system’s technical performance? (Risk Management 
Guide for DoD Acquisition, 2001, p. 19) 


Based on the assessments, the PMO may choose among 


several risk mitigation (i.e. handling) techniques. 





a. Risk Avoidance 
A PMO may avoid a risk of one alternative by 


choosing another, less risky alternative. This process is, 





in essence, a method to reduce risk since it does not 








completely eliminate risk. An important distinction to 
make is that risk avoidance must be a conscious decision to 
choose lower versus higher risk options. Avoiding risk by 
ignoring its presence and potential impact is an 


unacceptable solution. 


Risk avoidance may be done in parallel with “the 
up-front requirements analysis, supported by a cost per 
requirement trade study. The concept of Cost as an 
Independent Variable (CAIV) is an example of such a study. 
It is imperative that user representatives are present 
during any trade-off study or decision.” (Risk Management 


Guide for DoD Acquisition, 2001, p. 20) 
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Risk cannot be altogether avoided. Remembering 
that a risk represents an opportunity, risk aversion can 


lead to a poor managerial environment. 


A risk-averse culture inhibits risk management 
more than does the lack of a management 
infrastructure or a repeatable method. Such a 
culture generally rewards crisis management and 
punishes those who identify why the project may 
not succeed. (GSAM, 2000, p. 6-19) 














It is evident that the avoidance of one risk in 


favor of another, less-risky alternative is not the same as 








attempting to eliminate risk from a program altogether. 

b. Risk Control 

Risk may be controlled through the continuous 
monitoring and correction of risky conditions. Risk 
control “monitors and manages the risk in a way that 
reduces the probability and impact of its occurrence on the 
program.” (Risk Management Guide for DoD Acquisition, 2001, 


p- 19) Risk control involves reviews, inspections, risk 





mileston reviews, development of fallback positions and 





Similar management techniques. Controlling risk involves 








“developing a risk reduction plan and then tracking to that 





plan.” (GSAM, 2000, p. 6-20) The following lists examples 


of risk control actions: 











° Multiple development efforts 

° Alternative designs 

° Early prototyping 

e Incremental development 

° Technology maturation efforts. 

e Use of mock-ups, and 

° Modeling and simulation (Risk Management Guide 


for DoD Acquisition, 2001, p. 19-20) 
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While this is not an exhaustive list, it provides 
examples of risk control act ions while, again, not 


eliminating risk. All of these methods reduce unnecessary 











risks while working to meet user requirements. 
Cc. Risk Assumption 
Risk assumption involves a conscious decision to 


accept a risk level and potential impact of occurrence 





without taking any steps to manage or reduce the risk. The 





challenge for PMs lies in determining an acceptable level 





of risk. Risk assumption is best reserved for low-level 
risks, in terms of impact, or risks whose probability of 
occurrence is remote. Whenever possible, PMOs will handle 


risk assumptions by ensuring that a contingency plan is in 





place to address and handle emerging risks previously 
assumed in the program. A management reserve, additional 


funds, personnel or schedule time, must be in place to 





accomplish contingency management actions. (Risk Management 
Guide for DoD Acquisition, 2001, p. 21) 

d. Risk Transference 

Risk transference involves more than one entity 


haring risk, which is often cost risk. This technique is 





s 
frequently used between th Government and contractors. 
T 


he Government provides a contractor financial incentives 





(award fees, contractual incentives), for example, to share 


in managing risk. A contract between the Government and a 








rime contractor generally initiates the risk transference 





p 
process. The Government may provide financial incentives 
t 


Oo a prime contractor to minimize or reduce risks in 





numerous risk areas to include system technical 








performance, development cost and adherence to. schedule. 
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This thesis discusses risk transference through the 
contracting process in subsequent chapters. 

4. Risk Tracking/Monitoring 

Risk monitoring is the continuous process of “tracking 


and evaluating the risk management process by metric 














reporting, nterpris feedback on watch list items and 
regular enterprise input on potential developing risk 
areas.” (Systems Engineering Fundamentals, 2001, p. 139) 
The process involves evaluating how current and past risk 


handling actions compare with previously established risk 
management metrics. Program metrics are used for formal 
analyses of how well the various development processes are 


progressing in comparison to TPMs, schedule predictions, 





technology maturity, etc. 


The purpose of monitoring and tracking risk is two- 


fold. First, to determine if risk elements are in danger 





of adversely affecting cost, schedule or performance of the 


program. Second, risk monitoring aids in identifying risk 








areas not initially identified and assessed. The “Goal, 





Question, Metric paradigm” (GQM) is a simple example of the 


risk tracking/monitoring process. (GSAM, 2001, p. 6-21) 


The GQM method consists of the following steps. The 
first step is to select the goals of the risk area- 
monitoring program. The second step is to identify “the 


questions that should be asked to determine if the goals 





are being met.” (GSAM, 2001, p. 6-21) The final step is to 








identify metrics or indicators that allow one to answer the 





question, “Are the goals being met?” (GSAM, 2001, p. 6-21) 





The final step in the risk tracking/monitoring process 








is to document the findings. A program office-wide shared 
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database allows all PMO personnel to update risk area 
progress and emergent threats in the program. 
Documentation “provides the basis for program assessments 
nd updates as the program progresses.” (Risk Management 


uide for DoD Acquisition, 2001, p. 21) Proper risk 





a 
G 

documentation also helps to incorporate new personnel into 
the program office and reduces the hazard of repeating past 


istakes. Depending on the technical depth and size of a 


ocumentation to be presented at established intervals. 





m 
program, PMs will establish a standard list of risk 
d 
ak 


he following list illustrates example reports: 

















° Program metrics 

e Technical reports 

e Earned Value (EV) reports 

° Watch list 

° Schedule performance report 

° Critical risk process reports (Risk Management 


Guide for DoD Acquisition, 2001, p. 22) 
The above list provides examples of reports that may be used to 


document the implementation of the RMP to assess its successes 








or shortfalls. The next section will analyze several techniques 








that DoD can use to manage risk in developmental systems. 
D. DEPARTMENT OF DEFENSE RISK MANAGEMENT TECHNIQUES 
Many risk management techniques are available to the 


DoD. The DoD 5000.2-R requires PMs to ensure that 





contractors’ management information systems used in 
“planning and controlling contract performance meet the 


Earned Value Management Systems (EVMS) guidelines.” (DoD 





5000.2-R, p. 49) The Program Milestone Decision Authority 


(MDA) may waive the requirement, in some instances. Other 








than EVMS, no particular risk management technique is 
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mandatory in DoD. This section discusses several risk 
management options available to PMs. 


1. Test and Evaluation (T&E) 








Periodic Test and Evaluation (T&E) events early and 
throughout a program’s development are a method of 
evaluating the progress and technological maturity of a 


system and identifying new risk areas. A test plan is a 





Tp 


risk reduction method if implemented early. The Té&E 








process is “an integral part of the systems engineering 





process which identifies levels of performance and assists 





the developer in correcting deficiencies.” (Test and 


Evaluation Management Guide, 2001, p. 1-1) 





T&E is an important risk management technique because 
it helps developers and managers evaluate levels of 


technical performance, reliability, maintainability, 








technical maturity and cost and schedule conformance. T&E 


is a proactive measure that validates arned levels of 





performance and identifies emerging risks so they may be 


managed and tracked: 


Correcting defects in weapons has been estimated 
to add from 10-30% to the cost of each item. 
Such costly redesign and modification efforts can 
be reduced if carefully planned and executed test 
and evaluation programs are used to detect and 
fix system deficiencies.” (Test and Evaluation 
Management Guide, 2001, p. 1-1) 











T&E, though often costly and time-consuming to 
perform, has the potential to help control costs and ensure 
a desired level of system performance in the long run of 


the program. Figure 11 illustrates the relationship 





between committing program dollars to thoroughly test and 
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evaluate the system incrementally throughout its life and 


the life-cycle cost of the system. 





Figure 11. Life-Cycle-Cost Decision Impact and 
Expenditures, from (Test and Evaluation 
Management Guide, 2001). 








The figure demonstrates that a system that is not 
properly tested early during its life cycle may incur far 
greater Operations and Support (O&S) costs than a system 
that undergoes a thorough T&E plan to identify and manage 
risks early throughout system development and demonstration 


(SDD) . 





T&E also serves as a decision-making tool for senior 





leaders in DoD. T&E events are required before a system 





can undergo a Milestone Review. Figure 12 illustrates the 


© 


relationship between T&E and the Acquisition Process. 


The Test and Evaluation Master Plan (TEMP) is written 














as a part of the formal Acquisition Strategy pending a 


Milestone B decision authorizing entry into System 
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Development & Demonstration (SDD). The TEMP addresses 
system items to be tested as well as laying out the 
Integrated Test Program (ITP) Schedule. The TEMP is 
updated continuously throughout the program and officially 


at each Acquisition Milestone. 


Low Rate 
intial 
Production 





Figure 12. Testing and the Acquisition Process, 
from (Test and Evaluation Management Guide, 
2001). 





The TEMP updates include guidance from the MDA on 
testing areas of interest during the follow-on acquisition 
phase. Testing areas focus on validating system 
capabilities and detecting and reporting of “deficiencies 
that may adversely impact the performance capability or 
availability/supportability of a system.” (Test and 
Evaluation Management Guide, 2001, p. 1-4) 


In summary, T&E is “the discipline that helps to 


illuminate risk areas of vulnerability.” (Test and 
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Evaluation Management Guide, 2001, p. 1-7) A rigorous Té&E 
program can identify and manage program risks in a manner 


that saves time and money, while also ensuring that the 





tester provides the user timely and cost effective answers 








to operational requirements. 


2. Cost as an Independent Variable (CAIV) 





Cost as an Independent Variable (CAIV) is the process 


of balancing cost, schedule, performance and risk early in 








a systems development in order to manage a program to a 
cost objective.” (Risk Management Guide for DoD 
Acquisition, 2001, p. 29) CAIV involves a joint PMO-user 


representative trade-off analysis between system 





performance and program costs. The underlying premise of 
CAIV is that “if costs are too great, and there are ways to 


reduce them, then the user and developer may reduce 





performance requirements to meet cost objectives.” (Risk 


Management Guide for DoD Acquisition, 2001, p. 30) Risk 





assessments are essential in the CAIV trade-off process. 





Assessing risk areas and identifying cost drivers 
provide PMs and user representatives with data that can be 
used when conducting trade-offs between system performance 
and cost. The concept of CAIV is that “equal emphasis must 
be placed on managing cost and schedule risks” as it is on 
system technical risk. (Risk Management Guide for DoD 
Acquisition, 2001, p. 30) 

3. Earned Value Management (EVM) 








The Earned Value Management System (EVMS) is a joint 


DoD-Industry agreement established in 1995 that details DoD 








5000.2-R contractor requirements with respect to the 
implementation of Earned Value Management (EVM). EVM is a 
process that, through one hundred percent system 
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decomposition and definition, evaluates a program’s 
progress in terms of cost and schedule. EVM can be used as 


an “early warning signal” to a PM to identify risks of 





overrunning cost or schedule constraints. (Earned Value 


Project Management, September 2002) 


By decomposing a system’s requirements and defining 


the system thoroughly, managers can provide cost estimates 





per development function and track the program’s progress. 























PMs periodically assess actual costs to date versus 
projected costs and actual time requirements versus 
projected time to determine variances. The identification 
of variance can help identify new or underestimated risk 


areas and alert the PM to take action to assess. and 


mitigate the cost or schedule risks. Figure 13 illustrates 





that a program’s progress may be evaluated after only 15% 





completion of the program. 


The key to using EVM effectively is an accurate 


program process definition and decomposition. When used 








properly, EVM affords a PM visibility on a program’s cost 
and schedule status. The PM can then make necessary 


changes or perform trade-off studies to meet cost and 








schedule thresholds. EVM is an ffectiv technique to 





combat cost and schedule risks. 
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Figure 13. Earned Value Management As a Risk 
Management Technique, from (Earned Value Project 
Management, September 2002). 








4. Modeling and Simulation 
Modeling and simulation (M&S) may be used by a PMO to 


manage risk throughout the entire life cycle of a system. 





Models and simulations can “reduce time, resources, and 


acquisition risk” and may contribute to increasing the 





system’s overall quality and performance. (Risk Management 
Guide for DoD Acquisition, 2001, p. 27) In managing risk, 


MéS can assist in the following ways: 


e Develop alternate concepts during system design 

° Predict performance in support of trade-off 
studies 

e Evaluate system design and support preliminary 


design reviews 


° Predict performance and supplement live tests 
during system testing 








° Examine the military value of the item 
e Determine the impact of design changes 
° Hone requirements 
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° Develop life-cycle support requirements and 
assessments (Risk Management Guide for DoD 
Acquisition, 2001, p. 27) 


The risk management techniques this thesis. has 
addressed are not mutually exclusive. For, example M&S may 


be used extensively during Té&E. 














Modeling and Simulation during T&E may be used for 





“concept evaluation, extrapolation, isolation of design 


ffects, fficiency, representation of complex environments 





and overcoming inherent limitations in actual testing.” 








(Test and Evaluation Management Guide, 2001, p. 14-8) By 





performing M&S during T&E, the PM may thoroughly test a 


system under virtual conditions and environments, which may 





otherwise be cost-—prohibitive. MéS helps to reduce 
technical risk by discovering design effects on the overall 
system before physically incorporated into the system. M&S 


reduces schedule and cost risk by assisting engineers to 





make the right decisions early based on data gathered 





during M&S tests. Additionally, models, which prove to be 





accurate predictors of actual test events may allow the PM 
to waive further, live tests based on a high degree of 


confidence in the model’s data. 


Modeling and Simulation is only as good as the data 





and variables that are inputs to the model. MéS can be a 





great risk management tool if adequate time is taken to 
ensure the accuracy of input data. The DoD 5000.2-R 


encourages PMs to incorporate MéS activities where 





applicable to their respective programs because of the 





potential cost and time reductions as well as enhanced 


system development and performance validation. 
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Ss Including Risk Management in the Contracting 
Process 


The final risk management technique this thesis 
addresses is the inclusion of risk management throughout 


the contracting process. Managing risk through contracting 








often involves risk transference from either the Government 








to the contractor or vice versa. “By properly setting the 
expectations of all players, explicitly agreeing upon the 
deliverable items produced by the event, and securing 


sponsorship from project management, a high degree of 





success is assured.” (SEI, 1999, p. 17) The shift from 
Military Specifications and Standards to Performance based 
requirements placed increased risk in the hands of the 
contractor. However, “if a program fails because risk 


isn’t managed well by the contractor, the PM is ultimately 





responsible.” (Defense Acquisition Deskbook (DAD), Top 
Eleven Ways to Manage Technical Risk, 1998, p. 3-1) 


Numerous opportunities exist throughout the contracting 





process that enables the Government to manage system risk. 





The remainder of this section discusses risk management 
through the Request for Proposal (RFP) and contract award 
fee incentives. 

a. The Request for Proposal (RFP) 

Even before an RFP is released, a PM should 
conduct a preliminary risk assessment to ensure that “the 
program to be described in the RFP is executable within 
technical, schedule and budget constraints.” (DAD, Top 


Eleven Ways to Manage Technical Risk, 1998, p. 3-1) The 





RFP should require offerors to address their RMP and 





initial risk assessment and mitigation plan for moderate to 





high-risk areas. (DAD, Top Eleven Ways to Manage Technical 
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Risk, ~1993,;. px. 3-1) The RFP should also stipulate that 





offerors must make periodic risk assessment reports to the 
Government. Contractors’ reports serve as input to the 
PM’s risk monitoring and tracking program. By requiring a 
risk-based approach, offerors’ proposals should “state how 
they would plan and schedule [software] activities based 
upon realistic assessments of technical challenges and 
risks” so that the Government may evaluate management 
capabilities.” (GS AM, 2000, ps 6-30) Whether the 


development risk is hardware, software or a combination of 





both, the RFP is a vehicle to inject risk management 





activities into the program. The RFP contains several 
sections, which allow the Government to directly address 


risk areas in the solicitation. 


Section C, Description/Specifications/Statement 


of Work, includes any descriptions or specifications 





required in the offeror’s response. (DAD, Top Eleven Ways 
to Manage Technical Risk, 1998, p. 3-2) Example Section C 


wording that addresses risk is as follows: 


The Offeror shall describe its proposed risk 

















management program. The Offeror shall describe 
how they intend to identify, assess, mitigate, 
and monitor potential technical risks. Critical 
technical risks which may adversely impact cost, 
schedule, or performance shall be identified 
along with proposed risk mitigation methods for 
all risks identified as moderate or high." (DAD, 
Top Eleven Ways to Manage Technical Risk, 1998, 
DS -3=2) 


Section C allows the Government to evaluate 
offerors’ RMPs and compare competing contractors’ responses 


with respect to risk identification, assessment and 








handling. The contractor’s detailed Statement of Work 
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(SOW) submitted in response to the RFP can inform the 
Government as to the level of contractor requirement 


understanding. The SOW may also warn PMOs of high-risk 





acquisition plans on the part of the contractor. 





Section L, Instructions, Conditions, and Notices 
to Offerors, provides instructions to the offeror in 


proposal preparation. The Government may elect to include 





risk management requirements in Section L as long as the 
risk items are consistent throughout the RFP. Example 


Section L language is as follows: 


The Offeror shall discuss past/present 
performance in the implementation of risk 
reduction/mitigation efforts similar to those 
proposed for the reduction of all risks 
identified as moderate or high.” (DAD, Top Eleven 
Ways to Manage Technical Risk, 1998, p. 3-2) 








Section L ensures that offerors include 


information pertaining to risk that may enable’ the 





Government to evaluate a contractor’s technical past 





performance or ability to manage technical risk. 


Section M, Evaluation Factors for Award, notifies 
offerors of “the evaluation factors against which all 


TG 


proposals will be evaluated.” (DAD, Top Eleven Ways to 

















Manage Technical Risk, 1998, p. 3-2) Section M_ should 
focus on areas outlined for offerors in Section L so that 
the Government’s instructions for proposal preparation are 
consistent with the evaluation criteria. Section M should 
list the relative importance or hierarchy of evaluation 
criteria such as past performance, risk management, 
technical performance, cost, schedule, management, etc. 


The Government is not required to quantify the importance 
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or ranking of evaluation criteria, but must inform offerors 





which criteria will be considered most in the source 
selection process. An example Section M risk management 


language follows: 


The Government will evaluate the Offeror’s 
proposed risk management program and plans for 
identifying, assessing, mitigating, and 
monitoring risks, as well as proposed plans for 
mitigating those risks identified as moderate or 
high." (DAD, Top Eleven Ways to Manage Technical 
Risk, 1998, p. 3-3) 














Through specific risk management Section L 
instructions and corresponding Section M evaluation 


criteria, the Government can ensure that only proposals 





that demonstrate responsible and capable risk area handling 


are considered for contract award. 


The DoD 5000.2-R states that risk reduction 
through the use of mature technology will be a significant 
factor in the Source Selection Process (SSP). The purpose 


of the SSP and competition in contracting is to “select the 





contractor whose performance can be expected to meet the 
Government’s requirements at an affordable price.” (DAD, 


Top Eleven Ways to Manage Technical Risk, 1998, p. 3-4) 





Several vehicles, including the detailed Statement of Work 








(SOW), aid the PM in evaluating solicitation respondents. 





b. Statement of Objectives (SOO) 


The Statement of Objectives (SOO) is a concept 





that transfers the responsibility for preparing the 


Statement of Work (SOW) from the Government to the offeror. 





(DAD, Top Eleven Ways to Manage Technical Risk, 1998, p. 3- 
3) The SOO is the “primary document for translating 


performance requirements into contractual tasks.” (GSAM, 
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2000, p. 8-9) The SOO encompasses top-level objectives and 
allows offerors greater flexibility and creativity in 
responding with SOWs that are more detailed. SOOs are 
intended to not limit contractors by imposing restrictive 


specifications. 


Having offerors prepare SOWs is intended to 
reduce costs and time previously spent by the Government. 
The SOO may inform offerors of the Government’s objective 
of identifying, assessing, handling and tracking risk 
areas. Contractors may respond with detailed RMPs or risk 
assessment methodologies in the SOWs contained in their 


proposals. If identified in Section M, offerors’ risk 





management methodology contained in the SOW may be used as 


evaluation criteria during the SSP. 





Figure 14 illustrates the RFP preparation 


process. 


Work Specifications Selection 
Breakdown Factors 
Structure 
Statement Draft Solicitation 
a Request for Review 


Elements 


Objectives i enc 
Acquisition Contract Data 
Strategy Requirements cota 


List 





Figure 14. RFP Preparation Process, from (GSAM, 
2000). 
c. Risk Management Based Award Fees During 


Contract Administration 
The process of selecting a contract type is based 
on many factors. The contractor’s technical ability, 


urgency of the program or the type and complexity of the 
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program are all examples oof contract type decision 


criteria. Providing incentives for contractors to identify 





and handle risks through Contract Award Fees can be a 
valuable technique for the Government in systems 


acquisition risk management. 


Award Fee determinations may be based on analysis 


of offerors’ SOWs and “identification of critical areas of 





program risk.” (DAD, Top Eleven Ways to Manage Technical 
Risk, 1998, p. 3-5) Moderate to high-risk areas are 


generally good candidates for award fees becaus they 





encourage the contractor to focus specific attention on 


areas whose potential impact to the program is highest. 





Tying award fees to risk areas identified as moderate to 
high risk ensures both Government and contractor attention 
and communication. Award fee discussion between the 


Government and the Prime Contractor should be held 





regularly. Award fee discussions should be held quarterly 
or even monthly to ensure continuous performance feedback 
for the contractor. This process facilitates open and 
Frequent communication between the PMO and the Contractor 


and ensures that risk management is a continuous process 








that requires constant attention. 


In order to administer an award fee type 
contract, contractor performance and award fee criteria 
must be clearly articulated and measurable. Award fee 


periods are often tied to specific events such as 








Developmental Test and Evaluation (DT&E) events or 








Operational Test and Evaluation (OT&E) events where the 


contractor has an opportunity to demonstrate achieved 
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technological maturity or system integration. Award fees 


may be incorporated into several contract types. 


A Cost Plus Award Fee (CPAF) type contract is a 
“cost reimbursable contract that provides for aé_e fee 
consisting of a base amount fixed at inception of the 


contract and an award fee amount that the contractor may 








earn in whole or in part during performance.” (Bonneville 
Purchasing Instructions, 1998, p. 7-7) The program cost 
estimate is determined during Government —Contractor 
negotiations. An Award Fee Plan (AFP) is agreed to upon 
inception of the contract. The AFP lays out award fee 





periods as well as award criteria. 


The Government must provide the Prime Contractor 
with frequent feedback concerning contractor performance 
before it can award, reduce or withhold an award fee. This 
enables the PMO and the contractor to communicate 
regularly, and openly, concerning the progress of the 
program and the status of the program risk handling. 
Including risk management as part of award fee criteria is 
a method of transferring risk from the Government to the 
contractor. 

E. SUMMARY 

This chapter first defined risk and risk management in 
the DoD environment. Risk is the probability of an event 
occurring and the likely impact that event has on the 
outcome of a program. Risk management involves the 


identification, assessment, mitigation and finally the 





tracking and documentation of risk areas within a program. 


Several categories of risk, or risk areas exist in DoD 


acquisition. A program’s success is based on the system’s 
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ability to achiev desired performance characteristics 
within cost and schedule constraints. Risk areas, whether 


technical, management or requirements can adversely affect 





a program’s ability to comply with cost and _ schedule 





constraints and still meet expected performance 


characteristics. 


Finally, this chapter discussed risk management 
techniques commonly used in DoD systems acquisition. It 
introduced T&E, EVM, M&S, CAIV and the contracting process 


as areas with potential for managing risks. 
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III. THE ADVANCED AMPHIBIOUS ASSAULT VEHICLE (AAAV) 


A. INTRODUCTION 
This section provides background information on the 
Marine Corps’ Advanced Amphibious Assault Vehicle (AAAV). 


Subsequent chapters provide analysis of the AAAV program’s 





risk management techniques during the Program Definition 





and Risk Reduction (PDRR) Acquisition Phase. A Milestone 
III review is planned for FY 2007. 
B. AAAV HISTORY 

The Marine Corps has had an amphibian vehicle since 
1941. (Lissner and Dees, March 2002) Its presence in the 
Marine Corps arsenal has personified the Marines’ 
amphibious assault capability. The necessity for fighting 
effectively in the littorals has continued to validate the 


requirement. But, by 1992, the currently fielded AAV had 





surpassed its planned 10-year service life by twenty years. 


Three, separat Mission Area Analyses (MAA) were 





conducted to determine the AAV’s mission effectiveness and 


suitability. The result was the determination that the AAV 





was deficient in mobility (land and water), firepower, 
survivability and command and control. (AAAV DRPM, Program 
Overview, October 2002) The Center for Naval Analyses 





(CNA) conducted an AOA to examin alternatives for the 








Marine Corps and its amphibious assault capability. 





(Lissner and Dees, March 2002) 


The CNA presented thirteen different system solutions 
to the Marine Corps’ needs. The alternatives ranged from 
amphibian, swimming vehicles to Landing Craft, Air Cushion 


(LCAC) loaded Bradley Fighting Vehicles. Following the 
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CNA’s AOA, the Marine Corps Combat Development Command 
(MCCDC) conducted a supplemental analysis in 1995. The 





analysis yielded a “hands down” decision to pursue a “fast -— 
swimming amphibian.” (Lissner and Dees, March 2002) The 


criteria for analysis was based on cost, performance, 








mission effectiveness and total life cycle cost of the 





system. 


Following the 1995 AOA, the AAAV program entered 
Program Development and Risk Reduction (PDRR). The AAAV 
reached a Milestone II review in September 2000. A 
subsequent AOA was conducted following PDRR to determine 
the validity of the proposed concept and its projected 
costs (acquisition and life cycle), performance, and 


mission effectiveness. 


Figure 15. The Advanced Amphibious Assault Vehicle 
(AAAV), from (FAS, Military Analysis Network, 
October 2002). 


The analysis determined that the AAAV was, in fact, 
the system that the Marine Corps needed to overcome 


existing operational deficiencies of the AAV. The AAAV 





passed the Milestone II review in November 2000 and entered 


the System Development and Demonstration (SDD) phase. The 





AAAV is currently in SDD as of early 2003. (Lissner and 
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Dees, March 2002) (AAAV DRPM, Program Overview, October 
2002) 


The AAAV prime contractor is General Dynamics 
Amphibious Systems (GDAS) . Its subcontractors and 


respective subsystem are: 











e MTU: Engine 

e Allison: Transmission and gear boxes 

° Honeywell: Water jets 

e Ball: Antenna 

e CDC: Communications (AAAV DRPM, Program Overview, 


October 2002) 
The SDD contract calls for the production and testing 


of nine prototypes prior to the Low Rate Initial Production 





(LRIP) decision. As of October 2002, three prototypes have 
been tested in varying terrain and under’ different 


operational conditions. 











Prototyp 1 achieved the High Water Speed Key 











Performance Parameter (KPP) in April 2000 by reaching a 








maximum waterborne speed of 38 knots. Prototype #2 has 





conducted significant land mobility testing. The vehicle 
achieved Troop Carrying and Land Speed KPPs during 2000. 


The third prototype has undergone extensive land, water and 





mobility developmental testing including Early Operational 





Assessment (EOA) in 2001. (AAAV DRPM, Program Overview, 





October 2002) In the fall of 2002, the AAAV performed 
shipboard launch, recovery and interoperability testing. 
GC. AAAV CHARACTERISTICS 

The Marine Corps is developing the AAAV in response to 


noted operational deficiencies in the currently fielded 











Amphibious Assault Vehicle (AAV). The AAAV’s mission is to 
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“provide high speed transport of embarked Marine Infantry 
from ships located beyond the horizon to inland 
objectives.” (AAAV DRPM, Program Overview, October 2002) 


The AAAV will provide Marine forces with an enhanced 





capability to conduct ship-to-shore movement and greater 





mobility and speed once ashore. The AAAV’s range, speed 
and enhanced survivability will provide commanders with a 
“Multiple Options/Late Decision Capability.” (AAAV DRPM, 
Program Overview, October 2000) The following table 


illustrates AAAV’s improved capability over the existing 


AAV. 


System AAAV Requirement AAV Capability 
Function 


Cross-country 45 MPH (keep pace with | 15-20 MPH 
main battle tank) 


Troop-carrying | 18 combat-—equipped 18 combat—-equipped 
capacity troops troops 


Survivabilit (1) 4.5mm round w/out (1) 4.5 mm round 
enhanced armor plating | with enhanced armor 
plating 














NBC Protection | Overpressure System 

(crew and embarked 

personnel) 

Lethalit Defeat light armored Incapable of 
combat vehicle of defeating light 
2005-2025 time frame armored combat 
day and night on th vehicle with 40mm 
move and .50 cal weaponry 

















Table 1. AAAV and AAV Capability Comparison, 
from (Military Analysis Network/AAAV, October 
2002 and General Dynamics Land Systems, October 
2002). 





48 


The enhanced capability the AAAV provides the Navy and 


the Marine Corps is the ability to conduct Operational 





Maneuver From the Sea (OMFTS). Littoral regions previously 





denying maneuver forces of options will become maneuver 





areas and avenues of approach. The AAAV will extend the 
Marine Corps’ “operational reach.” (AAAV DRPM, Program 
Overview, October 2002) 
D. AAAV ACQUISITION AND PROCUREMENT 

The AAAV is the only ACAT ID program managed by the 
Marine Corps. The AAAV program office is unique as the 
Government and the prime contractor, GDAS, share the same 


facility in Woodbridge, Virginia. 


The Marine Corps will buy one thousand and thirteen 
(1,013) AAAVs to replace the AAV at a cost of nearly $7.6 
Billion. The AAAV’s Initial Operating Capability (IOC) is 
scheduled for 2008 with full fielding of the system 
beginning in 2012. (Military Analysis Network, 2002, p. 2) 





Figure 16 depicts the AAAV’s program schedule as of October 
2002. 
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Figure 16. AAAV Program Schedule, from (AAAV 
Direct Reporting Program Manager (DRPM), October 
2002). 


50 


IV. THE ADVANCED AMPHIBIOUS ASSAULT VEHICLE (AAAV) 
RISK MANAGEMENT STRATEGY DURING PROGRAM DEFINITION 
AND RISK REDUCTION (PDRR) 


A. INTRODUCTION 

This chapter presents the research methodology this 
author uses to address the primary and subsidiary research 
questions. The objective of this chapter is to provide 
data and information pertaining to the Advanced Amphibious 
Assault Vehicle’s (AAAV) Program Definition and Risk 


Reduction (PDRR) Phase risk management strategy. 


The AAAV Program Management Office (PMO) is employing 
several risk management (RM) techniques during the System 
Development and Demonstration (SDD) Acquisition Phase. The 
next chapter analyzes the lessons learned from the PDRR 


risk management strategy and how the lessons learned are 





helping to shape the risk management process during SDD. 
B. RESEARCH METHODOLOGY 


In order to obtain background information to answer 


the subsidiary thesis questions, the author conducted a 








thorough Internet search of Department of Defense (DoD) 
directives, manuals, guidelines, presentations, handbooks 
and reports as well as private sector risk management 
related sites. Additionally, the author conducted a 
literary search. The sources used in the presentation of 


the research included newspaper and magazine articles, 





books, trade journals and other library information 





resources. The author visited the AAAV Direct Reporting 
Program Management Office (DRPM) in Woodbridge, VA to 
conduct interviews with both Government and Prime 


Contractor program leadership and observe risk management 


Del: 


practices in use during SDD. The latter methodology 
provided great insight into the current DoD risk management 


environment. 


The majority of AAAV specific, PDRR and SDD Risk 


Management practices discussed in this thesis came from 





interviews with AAAV PMO department heads. The author 





interviewed the System Test Officer, the Senior System 


Engineer, the Deputy Director of Program Planning and 














Integration (PP&I) (also the Risk Management Coordinator), 
the System Chief Information Officer (CIO), and a support 
contractor charged with consulting on matters of risk. In 








addition to Government program office personnel, the author 








interviewed the General Dynamics Amphibious Systems (GDAS) 


Risk Coordinator (Program Planning Integrating IPT), 
Assistant Risk Coordinator, and a Quality Assurance 
Engineer. 





Cc. OBJECTIVE OF RESEARCH 





The objective of the research is to collect, decipher 





and present information and facts that lead to the analysis 
and discussion of DoD risk management practices. The 
vehicle for this presentation and analysis of data was a 


case study of the AAAV. The primary and subsidiary thesis 








questions are re-stated below: 
1... Primary 


e How have the lessons learned from the AAAV’s 
Program Definition and Risk Reduction (PDRR) Risk 
Management Strategy impacted the Program’s Risk 
Management Process during System Development and 
Demonstration (SDD) ? 


2. Subsidiary 








e What are risk and risk management in Department 
of Defense (DoD) systems acquisition? 
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e What techniques can DoD use to manage risk in 
developmental systems? 


e What is the AAAV program? 


e What are the lessons learned from the AAAV PDRR 
Risk Management Strategy? 





e What risk management approaches has the AAAV 
Program Office adopted to manage technical and 
programmatic risk during SDD? 


e What conclusions and recommendations can be drawn 
from this analysis? 


Chapters II and Til answered the first three 
subsidiary questions. This chapter presents data for the 
urpose of addressing AAAV risk management techniques 
uring PDRR. Subsequent chapters address the remaining 


uestions and conclude by answering the primary research 








uestion and providing conclusions and recommendations. 
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AAAV RISK MANAGEMENT TECHNIQUES DURING PROGRAM 
DEFINITION AND RISK REDUCTION (PDRR) 


This section addresses the following risk management 
techniques employed in the Advanced Amphibious Assault 


Vehicle (AAAV) Program during PDRR: 


e Information Technology Tools 





e The Joint Government —General Dynamics Risk 
Management and Resolution Process 


° Contracting 


e Government and Prime Contractor Co-Location and 
the IPPD process 








° Test and Evaluation 

1. Information Technology Tools 

The AAAV Team (Government and General Dynamics 
Amphibious Systems (GDAS)) utilized robust information 


technology tools to assist in the management of the program 


during PDRR. Information sharing within organizational 
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departments and especially across functional areas was 
crucial to the program’s successful risk management 
approach. The AAAV Team leveraged many tools and 
processes. This section discusses the Virtual Design 
Database (VDD) and the Virtual Integration and Assembly 


(VINTEGRA) systems. 





a. Virtual Design Database (VDD) 
The Virtual Design Database (VDD) is a tool that 


provides users with a virtual, integrated environment. 





Accessible to both Government and GDAS personnel at any 
desk or lap top computer, the VDD consists of user- 
friendly, windows-based electronic documents sorted by 


Integrated Product Teams (IPT) and aligned by the Work 

















Breakdown Structure (WBS). The VDD “consists of various 
distributed databases linked together via high speed 
network connections.” (Integrated Digital Environment 
(IDE), October 2002) IPT areas include Modeling and 
Simulation, Systems Engineering, Logistics, Mobility 
Products, Firepower Products, Integration and Assembly, 


etc. The VDD “provides AAAV IPT members with an on-line, 





real time, paperless communication system used to logically 
file and provide access to AAAV program documentation.” 


(IDE, October 2002) 


The VDD has the following characteristics: 


e Makes data available to all members, USMC, 
Government, and the subcontractors from a desktop 
platform 

° Provides desktop 3-D Visualization and _ solid 


model capability 





e Stores data in all electronic formats 
e Permits a document author or IPT lead to edit the 
document 
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° Integrated e-mail system 


° Fully integrated with other PC applications. 
(IDE, October 2002) 





VDD’s personal computer interface resembles a 








Work Breakdown Structure (WBS) format. Drop-down windows 
allow users to search files by IPTs. Risk Management is 
facilitated through features, or capabilities, built into 
the VDD. The features allow efficient sorting of risks. 


The VDD user is able to select a “Risk” window, which 





provides access to the entire risk repository containing 


individual risk forms. VDD also has a “Help” function, 











which assists unfamiliar users in finding documents or 


performing functions within VDD easily. 


Risk forms contain all information pertaining to 
particular risks within functional areas of the system. 


The names of the Risk Owners (to be discussed in the next 





section), risk assessments, status and mitigation 
activities are contained on risk forms. Users may review 


risks and status of mitigation actions as well as generate 








new risks. An automatic risk notification system alerts 
other functional areas and program leaders to emerging 
risks and changes in risk status by generating e-mails 
containing links to view electronic risk forms. Figures 


17, 18, 19 and 20 provide examples of VDD pages. 


55 





— 
baka: 
AAAV DEM/VAL CONTRACT COST 


COMPOSITION 


448 ae 


AAAV DEMONSTRATION VALIDATION 
SCHEDULE 





Figure 17. Virtual Design Database (VDD), from 
(Rob Kepner, EPMC Brief, 2002). 


Electronic Risk Forms allow VDD users to perform 


a myriad of functions. One may enter a new risk into the 





system, check the status of current risk mitigation actions 
on risks designated by specific risk numbers, update risk 
status, etc. Risk form editing is electronically 


restricted to the designated Risk Owners and others 





Government and contracting personnel specifically 


authorized to edit files. 


Figures 18, 19 and 20 illustrate the VDD’s access 
to risk forms and an example of a risk form as it appears 


on VDD. 
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Figure 18. VDD Path to Risk Forms, from (Rob 
Kepner, EPMC Brief, 2002). 





Figure 19. VDD Risk Form from (Rob Kepner, EPMC 
Brief, 2002). 
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Figure 20. Organization of Risks on VDD, from (Rob 
Kepner, EPMC Brief, 2002). 

b. Virtual Integration and Assembly (VINTEGRA) 

The AAAV Virtual Integration and Assembly system 
(VINTEGRA) is an engineering and manufacturing tool 
designed to support the continual refinement of system 
assembly and integration. VINTEGRA is a subset of VDD with 
many of the same characteristics: accessibility, personal 
computer interface, automatic e-mail notification, etc. 
VINTEGRA is intended to capture and facilitate integration 
and assembly (I&A) and production data. The data leads to 
the identification of risks associated with Ié&A and 
production. VINTEGRA, relying on software called 
ProProcess, provides shop mechanics with a paperless source 
of manufacturing and assembly instructions. VINTEGRA is 
located on AAAV’s Intranet. Shop mechanics use a standard 
Internet browser, Ls@u, Internet Explorer, to access 
VINTEGRA’s drawings and assembly instructions. VINTEGRA 


may be accessed through computer aided design (CAD) 
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workstations in the assembly area or on desktop computers 


anywhere on AAAV’s network. “Traditional ‘blue-print’ 





style line drawings of the design have been replaced by 





computer images rendered in the vividness of three 
dimensions (3-D).” (Defense Modeling and Simulation Office, 


1999) 


Multiple computer stations surrounding the 
vehicle hulls undergoing assembly are available to the shop 


mechanics for reference. VINTEGRA allows shop mechanics to 














electronically view the proper sequence and method of 


assembly prior to working on the actual hull. Mechanics 





control the speed at which they learn the self-paced 











assembly instruction. 


Design imperfections and changes are anticipated 








during prototype build. VINTEGRA is an interactive system. 
Mechanics may provide input to engineers if they encounter 
problems during assembly through an electronic problem 


reporting system. 








VINTEGRA contains an Electronic Problem 





Resolution System (EPRS) which allows for “real-time 
capture of assembly problems as they are discovered.” 


(Defens Modeling and Simulation Office, 1999) If a 





mechanic encounters problems applying assembly instruction, 


then he or she can mark up the Computer Aided Design (CAD) 





images to illustrate the exact location and nature of the 





discrepancy. Because VINTEGRA resides on AAAV’s Intranet, 





the mechanic’s corrections or indicated problem areas are 





immediately brought to the attention of engineering and 


design teams via electronic dissemination. The teams, in 








turn, may make real-time changes to the assembly process or 
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identify potential change requirements for the next design 


iteration via the configuration management process. 


The EPRS form contains the following major 


subject headings: 








° Problem Identification 

e Problem Description 

° Disposition of Short Term Fix 
° Verification of Short Term Fix 
° Disposition of Long Term Fix 

° Verification of Long Term Fix 


Shop mechanics can affect wide dissemination of 





the EPRS though VINTEGRA. 





The AAAV Team won the 1999 Defense Modeling and 
Simulation Office Award for VINTEGRA. 


2. AAAV Team PDRR Risk Management Process 





The program’s objective during PDRR is to continue to 
develop the design and engineering maturity of the system 
as well as to identify, assess and handle risks. Technical 
risks tend to attract the majority of attention during 


PDRR. 


The AAAV, Government program office contractually 
mandated that GDAS institute the program’s risk management 
process during PDRR. The AAAV Program Manager (PM) 
approved and oversaw the process. A risk management plan 
and its implementation by GDAS were second period award fee 
criteria during PDRR based on the Milestone II contract 


award. The next section addresses Risk Management through 





the contracting process in greater detail. 
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The AAAV Team’s risk management process involved five 


steps: 
° Risk Identification 
° Risk Analysis and Prioritization 
° Risk Planning 
° Risk Tracking 
e Risk Control (AAAV Independent Risk Assessment, 
March 2000) 
a. AAAV PDRR Risk Identification 
The first step in the AAAV risk management 
process during PDRR was Risk Identification. Any number of 


individuals or groups was empowered to identify risks and 
initiate the management process. Individuals, an IPT or 


any internal or external AAAV stakeholder could identify 





new or existing candidate risks. Existing risks could be 
transferred from one IPT to another depending on the nature 
of the risk and an IPT’s' expertise. New risks were 


initiated by sending an email or holding a conversation 





with an IPT lead or the program Risk Management Coordinator 


(RMC) . Existing risks were transferred by one IPT to 





another or by the Risk Resolution Board (RRB). (GDAS PDRR 





Risk Primer, April 1998) 


The Joint Risk Resolution Board (RRB) was a Joint 
DRPM/GDAS board designed to analyze and resolve risk issues 


that required senior DRPM and GDAS attention for 





resolution. 


At the RRB, IPTs presented risk summaries to the 


board members. Briefs highlighted risk area trends and 





emerging risks by IPT. The risks were categorized as High, 
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Moderate or Low-level risks. The AAAV Team categorized 
risks in the following manner: 


° High Risk (Red): Risk is most likely to cause 
major program impact/disruption. For these 
risks, a different approach may be required to 
successfully complete EMD. Priority management 
attention should be applied. 





° Moderate Risk (Yellow): Risk can cause som 
program impact/disruption. Risk can b 
resolvable during EMD by proper implementation of 
mitigation efforts that may involve a different 


e 
e 

















approach. Additional management attention may be 
needed. 

° Low Risk (Green): Risk is at an acceptable level 
with minimal or no known impact. (Independent 





Risk Assessment Team (IRAT), 2000) 





Each risk was listed by IPT name and by Risk 
Identification number as seen on the VDD for purpose of 
reference. (AAAV Program Office Risk Resolution Board 


Presentation, November 1997) 


The RRB had several objectives. First to provide 





senior leadership with decision support information and 


ensure cross-functional area communication. Decision 





criteria included the commitment of additional resources, 


acceptance of risk, trade offs and mitigation courses of 





action. Secondly, to provide a forum where decisions were 
made jointly between GDAS and the Government using the same 


data. Third, the RRB was the appropriate forum to close 








out risks nominated for this action. The RRB ensured that 


the risk management process was performed as intended. 








aastly, the RRB intended to reduce risk processing and 








handling cycle time to facilitate decision-making at the 


earliest possible opportunity. (Kepner, 2002) 
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Once a candidate risk was identified, the issue 
became an IPT or RRB agenda item. Doctrinally, IPTs 


reviewed candidate risks weekly and existing risks bi- 





weekly while the RRB reviewed program-level risk 
continually. The RRB attempted to meet formally on a 
monthly basis as a “decision-making forum.” (GDAS PDRR Risk 
Primer, April 1998) The RRB and IPTs reviewed candidate 





risks to determine one of the following courses of action: 











° Control the risk 
° Accept the risk 
° Reduce the risk 
e Transfer the risk (Kepner, 2003) 
Determinations on candidate and existing risks 
were made within five working days of identification and 


initial discussion. Candidate risks of minor severity were 





classified as Action Items. IPTs posted Action Items on 
the AAAV Action Item database found on the VDD. Hither the 


accepting IPT or RRB entered risks on the VDD accompanied 





by a mitigation plan. Figure 21 illustrates the AAAV, PDRR 





Risk Management Process with the Risk Identification step 


highlighted in gray. 


When a risk was identified and acceptance 
assigned, the process required further action. Formal 
assignment or assumption of Risk Ownership was required for 


each risk entered into VDD. 


The goal was to resolve and close risks at the 
lowest possible level. When a risk was discovered during 
PDRR, the Government and GDAS counterparts who uncovered 


and introduced the risk became “Risk Owners” for that risk. 





They jointly assumed responsibility for the risk from its 
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cradle to grave. Risk Owners were identified by name on 





all risk forms on the VDD to disseminate th overall 


responsibility for the particular risk. Risk owners were 





responsible for formally introducing new risks into the VDD 


through risk forms. 


Risk forms indicated the nature and severity of 
the risk as well as actions to be taken to mitigate the 


risks. 


The purpose of the risk forms was to provide an 
assessment of current risks and estimate risk resolution 
requirements and timelines. A critical function of the 


risk introduction, assessment and mitigation process was to 





identify resources required to mitigate risks. Risk Forms 


also contained fields for the entry of “Estimated Recovery 








Date” of the risk. Time estimates were designed to allow 








the program office to analyze variances in actual versus 





estimated risk mitigation times. 





While risks were jointly owned between GDAS and 
the Government, the Government assumed overall 
responsibility for the risk and its management. Should a 


Situation occur that a risk could not be resolved and 





closed at the lowest level possible, the risk was elevated 





through the IPT hierarchy to the Joint Risk Resolution 


Board (RRB) for resolution. 


The next step in the risk management process was 


Risk Analysis and Prioritization. 
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Figure 21. AAAV PDRR Risk Management Process: Risk 
Identification, from (GDAS PDRR Risk Primer, 
April 1998). 
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b. AAAV PDRR Risk Analysis and Prioritization 
The PDRR risk analysis and prioritization process 


began upon identification and acceptance of the risk. 





Within five working days, the owning IPT or RRB was 
required to initiate a Risk Management Form in VDD. Risk 
forms in VDD were matched with corresponding Work Breakdown 
Structure (WBS) numbers. The IPT or RRB assigned Keywords 
to risk forms. Keywords were “fields which enabled the 
user to specify unique terminology associated with the risk 


and aid in the early identification of trends, 








establishment of triggers and prioritization of constrained 


resources at the IPT and system level.” (GDAS PDRR Risk 





Primer, April 1998) Keywords were classified as product, 


practice or process: 


° Product Keywords: Names of mechanical, 
structural, software systems or subsystems unique 
to AAAV 

° Practice Keywords: Categorized into one of six 
classes: Acquisition, Design, Facilities, 
Logistics, Manufacturing and producibility or 


Test and Evaluation 





° Process Keywords: Categorized into six classes, 
which represent “DoD Best Practices”: Design, 
Test, Production, Facilities, Logistics and 


Management (GDAS PDRR Risk Primer, April 1998) 








The AAAV  RMC performed regular query-based 
searches of the database to identify trends in keywords and 


report the findings to the RRB. 





Risk analysis included the Risk Owner preparing a 
risk statement as a part of the VDD Risk Form. The purpose 
of the risk statement was to “quantify the cause of the 
risk and specify which requirement(s) cannot be satisfied 


if the risk is not mitigated.” The risk consequences field 
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on VDD served to “quantify the effect of degraded 
operational performance, increased cost, decreased 
reliability, schedule slip, etc. if the risk was not 
mitigated.” (GDAS PDRR Risk Primer, April 1998) The PDRR 


Risk Analysis and Prioritization process is highlighted in 





Figure 22. 

c. AAAV PDRR Risk Planning 

The third step in the PDRR risk management 
process was Risk Planning. Risk planning “includes all 


management aspects of dealing with risk by choosing a 





specific course of action for mitigation among the several 





alternatives available.” (GDAS PDRR Risk Primer, April 





1998) Risk planning included risk mitigation and risk 








tracking VDD entries. 


Risk mitigation plans included the following 








elements: 
° Probability of occurrence 
° Assessed risk level 
° Mitigation plan and history 
° Estimated recovery date 
° Risk tracking 
e Watch list inclusion 
e Date closed (GDAS PDRR Risk Primer, April 1998) 








67 


Known or 
Anonymous 
Individual Brings Up 
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Trends and 
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Figure 22. AAAV PDRR Risk Management Process: Risk 
Analysis and Prioritization, from (GDAS PDRR Risk 
Primer, April 1998). 
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The estimated recovery date was a mandatory entry 





and reflected the IPTs time estimate required to mitigate 


the risk. When the approving authority determined a risk 








to be fully mitigated or no longer relevant, then the risk 


was closed on VDD. The risk tracking entry allowed the 








risk owner to select between “Off-track” or “Monitor 


closely” depending on the severity of the risk and the 





progress of the mitigation efforts. “Off-track” 
highlighted a risk that was out of tolerance and “monitor 


closely” brought attention to a risk area that was 








approaching tolerance limits. Th categories allowed 


managers to prioritize risk areas. 








Included in the mitigation plan was the assessed 








risk level determined by the probability of occurrence and 





the severity of impact to the program. 


Risk mitigation plans included all activities to 





be conducted and resource estimates in order to mitigate 
the risk. All mitigation activities needed to be completed 
prior to the estimated recovery date. The risk’s history 
was updated each time an IPT revised a risk form. The 


mitigation history provided traceability. (GDAS PDRR Risk 





Primer, April 1998) 


Risks were assessed and categorized into facets 


of program impact: 


° Technical Performance: Risk associated with the 
enhancements of the design to maximize 
performance 

° Cost: Risks that impact budget 

° Schedule: Risks that impacts Milestones’ and 


Decision Points 


69 





° Programmatic: Risks related to resources (people, 


equipment, facilities, funding, etc.) and 
functions of the business 
° Supportability: Risks associated with fielding 


and maintaining the current 


system (DRPM AAA, 


Risk Mitigation Planning Guidance for the Risk 





Owner, 2002) 


The probability of the risks’ occurrence and the 


severity of impact on one or more of the above mentioned 





categories determined the overall 


Figure 23 depicts the AAAV PDRR Risk 





Moderate or High. 


Assessment matrix. 







What is the Likelihood 
Level the Risk will Happen 
a 


Remote 


a oOo 


Unlikely 
Likely 


Likelihood 


Highly Likely a 


Near Certainty 1 


COST SCHEDULE 


Minimal or Minimal or Minimal or 
No Impact No Impact No Impact 


<5% Additional Resources Acceptable with some 
Required reduction in Margin 
5-7% Minor Slip in Key Acceptable, with some 


Milestones: Not able 
to meet needed date Required 


reduction in Margin 


Major Slip in Key Acceptable. 


Milestones: Critical 
Path impacted 
Can't Achieve Key 
Team or major program 
Milestone 


No remaining Margin 


Unacceptable. 


Figure 23. 
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23 4 5 
Consequence 


TECHNICAL 
PERFORMANCE 


risk assessment: Low, 


HIGH - Unacceptable. Major 
[| disruption likely. Different 
approach required. Priority 


management attention 
required. 


MODERATE - Some 
disruption. Different 
approach may be required. 


Additional management 
attention may be needed. 





LOW - Minimum impact. 
Minimum oversight needed to 
ensure risk remains low 


PROGRAMMATIC SUPPORTABILITY 


Minimal or Minimal or 
No Impact No Impact 


Additional Resource Acceptable with some 
Required reduction in Margin 


Minor Slip in Key Acceptable, with 
Milestones: Not able reduction in Margin 
to meet needed date Required 


Major Slip in Key Acceptable. 
Milestones: Critical No remaining Margin 
Path impacted 
Can't Achieve Key 
Team or major program 
Milestone 


Unacceptable. 


AAAV PDRR Risk Assessment Matrix, from 


(GDAS PDRR Risk Primer, April 1998). 


d. AAAV PDRR Risk Tracking 
The RMC was largely responsible for risk tracking 
but relied on IPTs to continuously update risk histories 


and status. The RMC conducted database queries to search 





for trends in keywords. The RMC compiled the query results 
and presented them to IPTs and the RRB for analysis. 
Analysis sometimes included reallocation of resources to 
address emerging risks or trends. The RMC served as 
secretariat to the RRB in the risk presentation and 
analysis process. Figure 24 illustrates the PDRR Risk 
Tracking process. 

e. AAAV PDRR Risk Control 

Risk control was the day-to-day management of 
risks. (GDAS PDRR Risk Primer, April 1998) This phase of 
the risk management process included the synthesis and 
decision-making and execution of risk area courses of 
action. The IPT or RRB responsible for the risk determined 


one of the following courses of action in consonance with 








the RMC: 

e Close the risk and document lessons learned 

e Evaluate the need for re-planning strategies and 
system-level workarounds 

e Invoke alternative mitigation and contingency 
plans 

° Continue to track the risk and execute its 
mitigation 


The authority to make decisions on existing risks 
was based on the review and approval authority matrix shown 


in Figure 25. 
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Figure 24. 
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The formal PDRR risk management process began on 


conditions of a Cost Plus Award Fee contract at the 











beginning of PDRR. GDAS developed the risk management 
process and the Government approved it. The emphasis of 
the risk management process was on resolving technical 


risks during PDRR. 


The following section discusses risk management 


through the contracting process during PDRR. 


Origination Mitigation Plan Review Authorized to Closeout 





























Source and Approval Risk 
High A-Level IPT A-Level IPT Risk Resolution Board 
B-Level IPT A-Level IPT Risk Resolution Board 
C-Level IPT B-Level IPT Risk Resolution Board 
D-Level IPT C-Level IPT Risk Resolution Board 
Moderate A-Level IPT A-Level IPT Risk Resolution Board 
B-Level IPT A-Level IPT A-Level IPT 
C-Level IPT BHLEvel LPT B-Level IPT 
D-hevel. LPL C-level. LPT C-Level IPT 
Low A-Level IPT A-Level IPT A-Level IPT 
B-Level IPT Boleved. LPT B-Level IPT 
C-Level IPT C-Level IPT C-Level IPT 
D-Level IPT D-Level IfT D-Level IPT 
Figure 25. AAAV PDRR Risk Review and Approval 
Authority Matrix, from (GDAS PDRR Risk Primer, 





April 1998). 


3: Risk Management Through the Contracting Process 


This thesis introduced in Chapter II several methods 





to reduce risk through the contracting process. Examples 





include risk-based Requests For Proposal (RFP), weighting 


of source selection evaluation criteria, Statement of 





Objectives (SOO) language and contractual incentives. This 
thesis presents the AAAV program’s use of contractual 


awards to incentivize risk management. 
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Following the Milestone I decision, the Government 
awarded GDAS a Cost Plus Award Fee (CPAF) contract for the 
PDRR phase. The PDRR Statement of Work (SOW) required GDAS 
to have a risk management plan. (Kepner, 2002) The PDRR 
Second Period Contract Award Fee plan was tied to GDAS’ 
risk management performance. The DRPM AAA established 
performance criteria to evaluate GDAS’ risk management 
program and supervised the process implementation during 


PDRR. 


GDAS’ full award fee amount depended, in part, on its 
ability to implement an effective risk management program 
and manage risks during the period. 


4. Government and Prime Contractor Co-Location 





The AAAV DRPM and GDAS have co-located at the AAAV 


Technology Center in Woodbridge, VA and the Worth Avenue 








Technology Annex (WATA) in Dale City, VA three miles south 





of Woodbridge. Both facilities are two-story buildings 
with the DRPM AAA occupying the top floor and GDAS the 





bottom floor. Both facilities have assembly areas where 
AAAV command mock ups and personnel variant prototypes were 
assembled during PDRR and continue to be assembled in SDD. 


The AAAV Team’s co-location is truly unique in defense 








acquisitions. Government and Prime Contractor co-location 
is intended to enhance communications and reduce program 


risks inherent with limited cross-functional area 





interaction. One objective of co-location is to facilitate 





the Integrated Product and Process Development (IPPD) 








environment. Communication is crucial to successfully 


implementing an IPPD process. 
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a. Integrated Product and Process Development 
(IPPD) 


Chapter II of this thesis introduced Integrated 
Product and Process Development (IPPD). The DoD defines 





IPPD as “a management process that integrates all 








activities from product concept through production/field 


support, using a multifunctional team, to simultaneously 








optimize the product and its manufacturing and sustainment 


processes to meet cost and performance objectives.” 








(Department of the Navy Acquisition, March 2000) IPPD is a 


systems engineering process that incorporates the use of 








and interaction between multifunctional teams, or 


H 


ntegrated Product Teams (IPT). This section provides 


examples of IPT interaction in the IPPD process during 





PDRR. 


b. AAAV Integrated Product Teams (IPT) 





The AAAV program uses twenty-eight IPTs with 
“membership representing every stakeholder in AAAV, from 
the Marine Users, Government Civilians, Industry (Prime and 
subcontractors), up through the Office of the Secretary of 
Defense.” (Pollution Prevention: AAAV, October 2002) The 


IPTs are involved in every aspect of system development and 











the systems engineering process. The AAAV IPT hierarchy is 


made up of four levels of IPTs as indicated in Figure 26. 








Figure 27 depicts the overall AAAV IPT environment. 





LD 


Product 
Teams 
(Level D) 


Design 
Teams 
(Level B) 


AAAV System 
Team 
(Level A) 





Figure 26. AAAV IPT Hierarchy, from (Rob Kepner, 
EPMC Brief, 2002). 
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Figure 27. AAAV IPT Environment, from (DRPM AAA 


IPT Brief, October 2001). 
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Co-location allows IPTs to work in continual, 





close proximity with Government, user and contractor 


counterparts. Numerous risk areas emerge during prototype 





development and production. The opportunity for IPT 








members to work together and communicate across functional 


area lines on a daily, regular basis is critical to 





capturing and managing risks. 


In the area of Environmental Safety and Health 








(ESH), each IPT has a member who provides expertise and 
representation for ESH issues. ESH issues play a 
Significant role in the AAAV development. ESH issues 





represent risk to the program in the form of large disposal 
costs, potential, adverse environmental impact and safety 


of use concerns. The following section uses ESH to 





illustrate IPT interaction in the development of a complex 
system. 
c. AAAV Environmental Safety and Health Program 


The National Environmental Protection Act of 1969 





(NEPA) mandates Government consideration of environmental 





impacts imparted by system development, fielding and 


disposal. In Chapter 5, the DoD 5000.2-R states: 


The PM shall evaluate and manage the selection, 
use, and disposal of hazardous materials 
consistent with ESOH regulatory requirements and 
program cost, schedule and performance goals. 
(DoD 5000.2-R, April 2002) 











The potential environmental impact of an 
acquisition system can create risks to the program. Risk 
areas include cost (development, life cycle and disposal), 


schedule and performance risks. The effective management 








of ESH issues in a developmental weapon system is critical. 


dt 





ESH representation in the IPPD process is vital to 


addressing potential risks for trade off analyses. 





Throughout the AAAV’s early program definition 





and continuing through SDD, ESH issues have impacted the 
system’s design and planned production, fielding and 


disposal. ESH factors can present significant risk to the 





program should the system fail to meet EPA standards or 


generate excessive costs attributable to minimizing 





environmental impact during operation or disposal. The 





AAAV Team has worked together to ensure the optimal balance 





exists between the AAAV’s performance and System Lifecycle 


Costs. 


AAAV IPTs each included ESH representatives 





during  PDRR. The representatives were tasked with 








including ESH considerations during the system’s design and 





development trade off processes. The intent was to ensure 
continual consideration of potential environmental risk 


impacts to the program throughout the design and prototype 





manufacturing process. 

5. Test and Evaluation (T&E) 

During PDRR, the AAAV program office produced three, 
fully functional prototypes for Developmental Testing (DT) 


and limited Operational Testing (OT): Pl, P2 and P3. 





The DRPM AAA conducted extensive DT during PDRR. P2 
conducted 4854 miles of land mobility testing: equivalent 


to nine vehicle years. (DRPM AAA, November 2002). P2 was 





also used in an Early Operational Assessment (EOA). 


An EOA is a type of test “conducted prior to, or in 





support of prototype testing.” (Test and Evaluation 


Management Guide, November 2001) A combination of AAAV PMO 
78 








Marines and 5°" to 95‘ percentile users manned the vehicle 





during the assessment held in 29 Palms, California in 








October 2001. The purpose of the EOA was to identify 


necessary design and configuration changes to the vehicle. 





The PMO also conducted a gunnery EOA to assess the AAAV’s 





weapon and fire control systems. The EOA enabled the AAAV 





program to solicit feedback on the system from users at an 








early stage in the system’s development and testing. 


Pl and P3 were used primarily for water mobility 





testing. P3 also participated in weapon station and land 


mobility test events. 


Pl and P3 testing included vehicle transition from 
water to dry land and vice versa. The prototypes were 
tested in the high water mobility mode to assess the 
vehicle’s ability to achieve threshold high water speed 
requirements. The program office conducted informal user 
juries with the five Marines who performed as DT vehicle 
crews. The user juries were encouraged to provide 


continuous feedback to system design engineers throughout 





testing. 


In addition to land and water mobility testing, the 
AAAV prototypes underwent communication testing (with use 
of a vehicle mock-up), firepower testing, survivability 


testing, habitability testing and lifecycle support 








testing. 


Extensive survivability testing included mock-up and 
prototype test activities to evaluate the system’s armor 
protection and crew survivability. In addition, the PMO 


placed significant effort in developing and testing an on- 


719 


board automatic fire suppression system to increase crew 


survivability. 


Life cycle support testing consisted of logistics 


demonstrations, maintainability demonstration, mean time to 





repair (MTTR) demonstration, interactive electronic 
technical manual validation (IETM) and human factors 


engineering testing. (DRPM AAA, November 2002) 





Figure 28. AAAV Survivability Testing, from (AAAV 
Developmental Test Brief, November 2002). 


E. SUMMARY 

During PDRR the AAAV Team worked to identify, assess, 
mitigate and track technical and programmatic risks. The 
AAAV Team employed several tools and techniques to manage 


risk. 


General Dynamics Amphibious Systems (GDAS) developed 
information technology tools to assist in the risk 
management process during PDRR. The Virtual Design 
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Database (VDD) was an Intranet tool designed to facilitate 
communication within and across both Government and 


contractor functional lines. VDD contained a “Risk” 





section, which enabled IPT members to initiate, track, 
update, and disseminate risk items throughout the AAAV 


program office. 


Virtual Integration and Assembly (VINTEGRA) is an 
engineering and manufacturing tool that provides’ shop 


mechanics with 3-D views of system assembly instructions 





and processes. VINTEGRA has an Electronic Problem 


Reporting System (EPRS) that allows shop mechanics to 





provide feedback to system designers and engineers on the 
assembly process and component design. VINTEGRA provides 
real-time updates to technical data packages (TDP) shared 


by GDAS and the DRPM AAA. 


The AAAV Team encouraged system risk input at all 


levels. Individuals who identified risks became “Risk 





Owners”. Government and contractor counterparts shared 


risk ownership. 


Risk Owners could initiate, edit and communicate all 
subsequent inputs or changes to risk areas Online through 


the use of Risk Forms found on the VDD. 


Risks are ideally resolved at the lowest possible 
level in the IPT hierarchy. Those risks deemed significant 


program risks or those incapable of resolution at lower IPT 








levels were elevated to a Joint Risk Resolution Board 





(RRB) . The Government and GDAS leadership comprised the 
RRB. The purpose of the RRB was to provide the program 


leadership with decision support information and clearly 
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communicate risks across functional lines at the 


Integrating IPT level. 


The PDRR contract was a Cost Plus Award Fee (CPAF) 
type. The award fee assessment included the evaluation of 
GDAS’ risk management plan and its implementation. The 
Government used award fees as an incentive for the Prime 


Contractor to proactively manage risks. 


The AAAV Team is co-located in Woodbridge, Virginia in 


he AAAV Technology Center and the Worth Avenue Technology 





nnex; the facilities are approximately three miles apart. 


he DRPM AAA members and GDAS employees work in the same 


TF HH Pp 


uildings. The principal purpose of co-location is to 








encourage and facilitate continual communication between 





Government and contractor personnel and across functional 


lines. Co-location is consistent with the principles of 





the systems engineering and IPPD processes. 


Co-location enables IPTs to meet regularly without 





Significant travel requirements or disruption of other 








responsibilities in the program offices. This thesis 
illustrates the program use of the IPPD process through a 
case study involving environmental safety and occupational 


health issues. 


Three AAAV prototypes underwent extensive 
developmental and early operational testing during PDRR. 
The purpose of the PDRR test plan was to identify 
performance risk areas and solicit user input. The PDRR 


test plan included logistics and life cycle testing. 





The next chapter provides analysis of the PDRR risk 


management plan. This thesis discusses lessons learned 
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from the PDRR phase and analyzes how those lessons learned 
have helped shape the AAAV risk management strategy during 
SDD. 
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V. ANALYSIS OF THE ADVANCED AMPHIBIOUS ASSAULT 
VEHICLE (AAAV) RISK MANAGEMENT PROCESS DURING 
PROGRAM DEFINITION AND RISK REDUCTION (PDRR) 


A. INTRODUCTION 


This chapter analyzes data presented in the previous 





chapter. In Chapter IV, this thesis presented elements of 
the Advanced Amphibious Assault Vehicle (AAAV) risk 
management program during the Program Definition and Risk 
Reduction (PDRR) acquisition phase. The data was 
categorized into five areas of research: 


e AAAV Information Technology Tools 





e The Joint Government —General Dynamics Risk 
Management and Resolution Process 


° Managing Risk Through the contracting process 


e Government and Prime Contractor Co-location and 
the IPPD process 

















e Test and Evaluation (T&E) 
B. RESEARCH METHODOLOGY 


This author’s research methodology involved extensiv 








Internet and literary searches. There are abundant 
resources available to research the Department of Defense 


(DoD) risk management environment and practices. However, 





with the cancellation of the DoD 5000 Series of documents, 





further research is necessary in the future to ascertain 


the revised risk management directives and methodologies 





used in DoD acquisitions. As this thesis is a case study 





of a developmental program that began under the DoD 5000.1 


and DoD 5000.2-R directives, the associated DoD risk 





management practices ar vident throughout the program’s 


history from PDRR through System Development and 








Demonstration (SDD). What will be of interest is how the 
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revised acquisition guidelines impact current and future 
risk management processes in developmental weapon systems 


including the Advanced Amphibious Assault Vehicle (AAAV). 





This author visited the AAAV program office in 
Northern Virginia. The ability to observe program 
operating procedures and interview both Government and 


contractor personnel was invaluable to this thesis. 





Cc. OBJECTIVE OF RESEARCH 

The purpose of this chapter is to analyze the risk 
management techniques the AAAV Team used during PDRR. 
Based on the analysis of the PDRR risk management process 
and techniques, this thesis intends to tie correlations 
between lessons learned from PDRR to the risk management 
process currently being used in SDD. 


D. ANALYSIS OF RISK MANAGEMENT TECHNIQUES USED DURING 
PROGRAM DEFINITION AND RISK REDUCTION (PDRR) 


The purpose of this section is to analyze the risk 
management techniques and processes the AAAV Team used 


during PDRR. This chapter addresses risk management 





process changes from PDRR to SDD, the system’s current 
acquisition phase. This chapter analyzes data in the 
following sequence: 


e AAAV Information Technology Tools 








e The Joint Government —General Dynamics Risk 
Management and Resolution Process 


° Managing Risk Through the Contracting Process 


e Government and Prime Contractor Co-location and 
the IPPD Process 











e Test and Evaluation 
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de Information Technology Tools 

The AAAV Team’s use of information technology (IT) 
tools during PDRR was invaluable. The joint Government — 
General Dynamics Amphibious Systems (GDAS) IPPD process was 


enhanced by the use of both the Virtual Design Database 





(VDD) and the Virtual Integration and Assembly (VINTEGRA). 


Recognizing the “communication multiplier” effect of tools 








like VDD, AAAV is developing a Similar, upgraded 
application. (Kepner, 2002) The following sections discuss 
the lessons learned and SDD perspectives of both VDD and 
VINTEGRA. 

a. Analysis of the Virtual Design Database 

The Virtual Design Database (VDD) provided the 


AAAV program office with a top-level view of the overall 











program. In particular, VDD provided a central repository 
for risk forms containing identification, assessment, 
mitigation planning and flexible documentation. VDD’s 





organization matched the system’s Work Breakdown Structure 
(WBS) from which the IPT organization was aligned. VDD was 


a logical, Lotus Notes-based application that facilitated 





involvement at all levels in the risk management process. 


VDD’s built-in help function facilitated risk 
data entry. Significant emphasis was placed on making VDD 
user-friendly in order to avoid discouraging IPT members 
from using the system. VDD’s automatic, interactive risk 
notification system ensured widest dissemination of risk 


forms and updates throughout the program offices. The 





automatic notification system was interactive because it 





required the determination of candidate risks to be made 





within five days of identification and electronic 





notification. IPT leads and program leadership were 
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involved in risk identification, mitigation plans and 
tracking from the outset of new risks. VDD helped to 


reduce the occurrence of poor communication within the 





program office. VDD applications helped to implement the 


risk management process. 





Risks identified, assessed and entered into the 


VDD were required to contain mitigation plans. The plans 





included estimated recovery dates and anticipated resources 





required to mitigate the risk. Mitigation activities were 





tracked through the VDD as described earlier in this and 








the previous chapter. 


To manage technical risks, the AAAV Team 


conducted extensive Modeling and Simulation (M&S), Cost as 





an Independent Variable (CAIV) trade-off analyses, and 
advanced technology demonstrators (ATD). The purpose of 
the ATDs was to evaluate the military utility and 
effectiveness of advanced technology concepts and _ to 
prepare to transition capabilities into the acquisition 


cycle. 
The AAAV PMO used ATDs and M&S extensively during 


PDRR to manage technical risks and perform trade off 


studies. The AAAV PMO built a 4/5-size hydrodynamic test 





rig to prove the planning craft technology as well as an 





automotive test rig to prove the vehicle’s retractable 
suspension and lightweight track technology. (Kepner, 2003) 
In M&S, the PMO used the NATO Reference Mobility Model 





(NRMM) using AAAV vehicle characteristic inputs (approach 





angle, departure angle, weight, height, etc.) to simulate 
vehicle terrain handling and mobility characteristics. 


Both ATD and M&S activities wer intended to greatly 
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mitigate technical risks at the outset of PDRR_ by 
identifying key system characteristics and identifying 


potential risk areas. 


The result of the risk management efforts during 
PDRR was the production of three fully functional AAAV 
prototypes. (Kepner, EPMC Brief) VDD was a way of 
communicating the results of trade-off analyses and other 


risk mitigation activities. 


The VDD was an effective risk communication tool 


during PDRR. One important characteristic of VDD was the 
“write once, read many” quality of the application. 
(Kepner, 2003) “Write once, read man” illustrates VDD’s 
utility as a communications multiplier within the program 











office. However, GDAS and AAAV Government personnel feel 








that the system was out-dated and incapable of being 
effective during SDD. Some program office personnel 
considered VDD a risk repository that eventually became 
saturated with data. (Rose, 2002) VDD lacked functional 


aspects that the program office feels are important during 





SDD: trend analysis, metric reporting capabilities, and 


analysis of variance (ANOVA). GDAS felt that VDD risk 








mitigation plans were stand-alone documents, not fully 





integrated into the overall program management. The 





Independent Risk Assessment conducted prior to Milestone II 
found that VDD was “more of a status report of activities 
or a list of planned events or meetings” rather than useful 


mitigation plans. (IRAT, September 2000) 


Subcontractors had limited access to VDD. High 


and moderate risks owned by sub-contractors were 
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incorporated into VDD. As the AAAV program grew, the need 





for an expanded system became evident. 


In SDD, the AAAV is developing an upgraded 


application that will replace VDD. The new system, Life 








Cycle Information System (LCIS), is a web-based application 





that will allow all Government, Prime and Sub-contractors 





to access the database from AAAV desktop machines as well 





as remotely through standard Internet browsers. Another 


impetus behind the development of LCIS is that the AAAV 





program has been designated as a Program Management Office 


of Life Cycle Support (PMOLCS). A PMOLCS designation 





directs that a PMO maintain responsibility for the system 
from “cradle to grave.” (Kepner, 2003) LCIS will be better 


equipped to support this function than was VDD. 


LCIS training is scheduled to begin in January 
2003. The AAAV Team recognizes the importance of 
incorporating sub-contractors in the risk management 
process. AAAV intends to fully train sub-contractors on 
LCIS and include them in the risk management process by 


offering this web-based application. 


GDAS, through their sub-contractor, Computer 
Systems Corporation (CSC), is developing LCIS to become a 
“customized document management system.” (Rose, 2002) LCIS 


will be capable of conducting ANOVA when = analyzing 




















mitigation efforts. LCIS will track projected resourc 
outlays versus predicted outlays; determine the 
effectiveness of the mitigation plan; and alert program 


leadership to cost, schedule or programmatic impacts. The 





AAAV Government team feels that LCIS will be able to 
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support a life cycle support system that emphasizes 
program-wide impact of risks. 


b. Analysis of Virtual Integration and Assembly 
(VINTEGRA) 


The Defense Modeling and Simulation Office 
awarded the AAAV Program Management Office (PMO) the 1999 


award for excellence in Modeling and Simulation. The 





benefits gained from VINTEGRA during PDRR are already 


evident in SDD and anticipated during subsequent production 





activities. 


Refining engineering assembly knowledge benefited 





U 


DRR efforts by accomplishing “scope commonly done during 





ra 


ngineering and Manufacturing Development (E&MD) (now SDD) 





hase.” (Defens Modeling and Simulation Office, 1999) 





p 
Furthermore, the “information captured using VINTEGRA in 
P 

















DRR, Significant data for production line analyses will be 
collected before the conclusion of PDRR.” (Defense Modeling 
and Simulation Office, 1999) The data collected during 








prototype builds in PDRR will be applied to SDD vehicles. 


The SDD prototype vehicle assembly process 
benefited from the use of VINTEGRA during PDRR. The 
assembly process has been refined and the Government and 
contractor have identified and corrected many manufacturing 


and produceability issues during the first three, PDRR 





prototype builds. Both the Government PMO and _ GDAS 





Bal 


consider production to be low risk partly due to VINTEGRA 


and the prototype assembly process. 


The process sheets in VINTEGRA are continually 





refined and updated. In essence, the manufacturing process 


during PDRR and for prototype builds was a prototype of the 
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eventual manufacturing process. The goal was to reduce 
risks associated with system manufacturability, such as 


schedule and cost risks. 


Shop mechanics’ familiarity with vehicle process 





sheets and th Internet-based browser and hyperlinks in 








VINTEGRA is expected to result in a learning curve effect. 





The benefits from the learning curve will be realized in 


the form of reduced vehicle build-time, or cycle time, and 











improved quality in manufacturing. The result will be 
reduced risk of missing system delivery dates and costly 


re-work of the system at the end of the production process. 


Additionally, the re-use of VINTEGRA’s’ three 





dimensional drawings and process sheets are expected to 





avoid costs normally incurred in th development of 





Interactive Electronic Technical Manuals (IETM). The 
electronic process sheet drawings have been validated and 


will be used in the IETM development. 


Electronic tools like VINTEGRA show potential for 





“shortening acquisition lead time and meeting war fighter 





needs faster, better, and cheaper, with the consequence of 
lower risk to the program.” (Defense Modeling and 


Simulation Office, 1999) This will enable the Marine Corps 





to field a higher quality system to the war fighter at a 





lower design to unit production cost (DTUPC). VINTEGRA has 
enabled the AAAV’sS Integrated Product and Process 





Development (IPPD) to anticipate system changes early in 


the vehicle’s prototype design and manufacture. 





Implementation of necessary design changes early in the 


system’s development will reduce technical and integration 
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risks and the risk of increasing DTUPC and delayed system 


delivery. 


Realizing the success earned through the use of 
information technology (IT) tools during PDRR as well as 
the need to expand existing capabilities, the AAAV program 
continues to innovate acquisitions through its creative use 


of IT applications during SDD. 


The Department of the Navy (DON) Electronic 





Business (eBusiness) Operations Office sponsored the DRPM 





AAA office to conduct a pilot program to pursue expanding 








VINTEGRA and, eventually, LCIS connectivity to remote 
sites. Sites such as remote test facilities did not 
previously have access to the information and communication 
capabilities captured by the AAAV’s IT applications during 
PDRR. AAAV’s creative IT innovations will continue to 
expand benefits in managing program risks during SDD by 


creating “virtual, integrated environments.” (Hepler, 2002) 





2. Analysis of AAAV PDRR Risk Management Process 
As discussed in the previous section, the AAAV PDRR 





risk management process relied heavily on the VDD to 
initiate, assess, communicate and track risks. However, 
VDD was only a tool to help execute the formal risk 


management process. 


The PDRR risk management process involved five steps: 


° Risk Identification 

° Risk Analysis and Prioritization 
° Risk Planning 

° Risk Tracking 

° Risk Control 
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This section analyzes the PDRR risk management process 
and discusses the lessons learned. The lessons learned 
from the PDRR risk management process helped to shape the 
proposed Government/GDAS risk management processes for SDD. 
At the time this thesis is being written, the formal SDD 
process is not yet implemented. This thesis briefly 
discusses the interim risk management process. The impetus 


of the risk management process is transitioning from 








technical to system-level integration risks. However, the 





relevance of this analysis is in the discussion of what 


worked and what did not work during PDRR and how those 





lessons learned are put to use during SDD. 
a. AAAV PDRR Risk Management Process 
The AAAV  PDRR risk management process was 


a 


effective. The primary goal was to identify, assess, and 


mitigate technical risk areas to support decision-making 





and design trade-offs. By focusing on technical risk 


management, the AAAV program was able to conduct CAIV 





analyses and perform modeling and simulation activities to 











determin the most effective design characteristics with 








respect to cost, schedule and performance. Key elements to 
the PDRR risk management process were ease of 


implementation via the VDD and effective communications of 





the process. 


The AAAV Team communicated the formal risk 
management plan through the IPT levels via a Risk Primer. 
The Risk Primer provided risk management process 
instructions intended for use as a desktop reference for 
risk management. The Risk Primer was an easy to use, 
procedural nineteen-page document. The intent was to 


introduce and include risk management in the normal 
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operations of the program office, not to bog IPT members 
down in formal methodology details. The primer provided 
instructions on identification, analysis, planning, 


tracking and control processes and use of the VDD. 


The lesson learned was that a simplified risk 





management methodology education process is crucial to the 
successful implementation of a risk management plan. The 
process and education, or training, must extend to the 


subcontractors as well as the prime contractor and 





Government personnel. As the program transitions from 





development to production preparation, ast becomes 


especially important to maintain a formal risk management 





process with all program stakeholders. Once the SDD 
process is formalized, the program intends to incorporate a 


risk primer as part of the process training. 


The methodology the AAAV Team used to manage 





risks during PDRR was sound. Because the majority of risks 


during PDRR were technical risks, the DRPM AAA RMC was a 





systems engineer with a part-time risk focus. The RMC was 


well equipped to coordinate and manage risk areas because 








of the integral role that he played during the acquisition 
phase. As the AAAV program transitioned from PDRR to SDD, 


the risk management coordination also transitioned. 


Additionally, shortfalls and limitations of the 
VDD are currently being addressed through the development 
of LCIS:. Currently, the AAAV Team uses. shared drives 
available on all desktop computers connected to the 
program’s Intranet. IPTs use standardized risk forms to 
initiate the risk management process. The risk forms used 


in SDD differ from those used in PDRR. 
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The SDD risk forms are Microsoft Word documents 
that capture the risk assessment and other pertinent 
information, as a placeholder pending the full 
functionality of LCIS. Figure 30 illustrates the current 
risk form the AAAV Team uses during SDD. Risk forms 


contain information on the following: 








° Risk Ownership 

° Risk Title and description 

° Risk Assessment 

e Risk Mitigation Plan and status 


The current risk forms and their location on the 
DRPM AAA/GDAS shared drives are temporary while the program 


migrates from VDD to LCIS. Risk owners are responsible for 





updating risk mitigation activities and assessments just as 





in PDRR. DRPM AAA and GDAS personnel may access and review 


all risk forms contained on the shared drives. Figure 29 





depicts an example AAAV SDD risk assessment form. 


While the AAAV program’s PDRR risk management 
process resulted in the successful deployment of three 


prototypes used in testing and production refinement, the 





PMO now benefits from lessons learned during PDRR. 
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AAAV Risk Assessment Form submitted by pt): Software 


Risk #: 69 (VDD = D-SFT-RSK-013) Risk Owner: C. Meconnahey Phone: x7528 Date: 10/30/01 
Risk Title: Ability to Accurately Capture and Trace SRS-level Requirements in a Parallel Development Effort Last Revised: 01/24/02 


WBS: 1.01.15.00.00 Applies to: SDD(EMD) _X___LRIP PROD 2 


DESCRIPTION OF RISK 


Condition: The parallel software development effort (Ada and RoseRT) in SDD Block 1 Phase Build 6.X necessitates capture 
and tracing of SRS-level requirements using separate methods. 


Likelihood 


Consequence if Realized: Tracking and tracing SRS requirements using two different methods can produce unpredictable end 
results. 

Context: The introduction of Object-Oriented Analysis and Design and a new supporting toolset (Rational RoseRT) within the 
AAAY Project has precipitated the need for a parallel development effort for the early phase the SDD Build 6.0. This parallel ee eee eee meee 
effort, however, poses a risk to requirements management for Build 6.0. ‘Consequence 
requirements will need to be accurately captured and traced using separate methods. The current method of captur ‘Giiseniieincs DeIVEE) 
“Shall” statements and tracing them to the appropriate RDDs will be utilized for the Ada development effort. The Woodbridge Place tm fughest one (3) 
facility will use RoseRT to produce models from the RDD and/or Use jevel requirements and trace RoseRT model elements Tech Sched. Cost 
(capsules, classes, etc.) back to the appropriate RDD or Use Case requirements tagged in the Rational RequisitePro SDD Block 

1-6X Project. 


Risk Mitigation Plan (Jmpiementation plan may be provided as an attachment) 


Mungatign Tene ( etige ERent [Start | Finish | (tated. cic.) _| (Estimate risk at completion) 


1. Create one SDD Block 1-6X ReqPro project to capture SDD requirements iia 
(ORD, S/SS, EFD, RDD, Use Case, Rose Model link) Sols MtBD, 


T_Capture SRS-Tevel “shall” statements Tor the Ada effort manually, using The broues 
same method as used for the PDRR Phase. eres 
—- : 


Here’s a 
sample Risk and 
Mitigation Plan. 





Figure 29. AAAV SDD Risk Assessment Form, from 
(AAAV Risk Mitigation Planning Guidance for the 
Risk Owner, November 2002). 





Those lessons learned are already helping shape 
and improve the risk management process during SDD. The 
next section discusses several lessons learned from the 
PDRR risk management process and introduces what practices 
the program has adopted as a result of the lessons learned. 

b. AAAV SDD Risk Management Process 


The goal of an effective risk management process 
during SDD is to deliver: 
° Fewer unexpected costs 
° Better cost control 
e Improved adherence to program schedule 


° Enhanced vehicle performance (Risk Coalition Team 
Brief, June 2002) 
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The current, SDD contract requires GDAS to 


develop for the Government’s approval a formal risk 











management process. The process used during PDRR focused 
primarily on technical and system-level risks. When 
consistently implemented, the PDRR risk management process 





ffectively helped the AAAV Team to manage technical risks. 
Similar to the PDRR phase, the GDAS SDD risk management 








plan will focus mainly on technical and system-level risks: 





areas that are within the scope of their contractual 


effort. 


However, the DRPM AAA recognized that Government -— 
specific risks exist in SDD that are not adequately 
addressed or managed through a risk management process 


developed and implemented by the Contractor like that used 





in PDRR. The types of risks in SDD are not the same types 
of risks encountered during PDRR. There are more 
programmatic risks, which are not within the scope of GDAS’ 
effort. Accordingly, the DRPM as well as the GDAS risk 


management processes needed to adapt to the program 








conditions. 


As a result, the Government is taking a more 





active leadership role during SDD to manage Government — 





specific risks separately from system-level, technical or 





developmental Contractor risks. DRPM AAA is modifying 


their risk management plan to nhance th focus on 





Government-specific risks such as funding, testing and life 





cycle considerations: those risks outside GDAS’ “sphere of 
influence.” (Risk Coalition Team Brief, June 2002) 


Meanwhile, GDAS is contractually obligated to develop and 











implement a risk management process for Government approval 
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and oversight through production. The portion of the 


Government’s plan for technical risks will leverage off of 





GDAS’ plan since both the Government and GDAS will actively 





manage the technical risks throughout the SDD phase. GDAS’ 





risk management is, again, embedded in contract award fee 


criteria during SDD. 


A lesson learned from PDRR is that risk 
management is a significant, continuous effort that 
requires “motivated personnel coordinating risk.” (Kepner, 


EPMC Brief) Additionally, the frequency of risk management 





meetings must coincide with program activities and the need 





for increased attention. For example, during the whole 








systems and subsystems trade processes, it is critical that 
risks be understood and managed as the system parameters 


and component capabilities are established. (Kepner, 2003) 





The risk management process, once approved and implemented, 





must be sustainable during all periods of SDD. 


The Independent Risk Assessment Team (IRAT) 
conducted prior to the MSII decision indicated that formal 


risk management practices waned during periods just before 





significant program events (i.e., prototype roll out, major 


test events, etc.) during PDRR. Programmatic focus on 





achieving key milestones temporarily impacted “systems 





engineering processes” (Kepner, EPMC Brief) Systems 
engineering processes include risk management, requirements 
traceability, synthesis, etc. The IRAT recommended that 


process sustainability be a key characteristic of the SDD 


risk management process. Accordingly, the DRPM AAA 





directed that a program directorate assume risk management 
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process coordination consistent with the activities 


normally conducted during SDD. 


The AAAV Team formalized a functional, program 
office directorate called Program Planning and Integration 
(PP&I). PP&I is responsible for the AAAV risk management 
plan and its implementation during SDD. The Program 
Integration Division Head is part of PP&I and is the 
program Risk Management Coordinator (RMC). Currently, both 
Government and GDAS members are refining the formal SDD 
risk management plan. The Government side of PP&I also 


focuses on Government-specific risks and advises the AAAV 





Program Management Team (PMT). The PMT consists of the PM, 


Deputy PM and Government Department Heads (i.e., Test and 











Evaluation Head, Manufacturing Head, Lead Engineer, PPé&I, 


etc.). 


The RMC chairs a bi-weekly meeting to address 


programmatic issues at the Department Head level called the 











Program Management Team (PMT) II. One of the focal points 
of the PMT II meeting is cross-functional risk 
communication and courses of action development. The 





purpose is to enhance communication of risks at the action 


officer level in the program and support decision-making at 








the next highest level, the PMT level. The PMT II advises 


the PMT on risk mitigation alternatives and resource impact 








for decision-making support. 





The PMT II is well suited to analyze system-wide 
metrics to assess the success of mitigation activities. 
Additionally, the meeting is used as a venue to discuss 


award fee determinations and recommendations for upcoming 





contract performance assessment reviews. 
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The need for the PMT II is essential during the 
SDD stage when the program transitions from development to 
production activities. The increase in the number of 
program personnel makes communication even more critical. 
Additionally, the PMT II can perform many of the same 
functions that the Risk Resolution Board (RRB) did during 
PDRR without some of the drawback encountered with the RRB. 





The RRB was an effective forum to analyze risk 


areas and serve as a decision-making board at the highest 





program office level during PDRR. The RRB process ensured 





dissemination of risks throughout the program from D Level 
IPTs to the PMT. However, amid the RRB’s successes, two 


challenges eventually emerged that caused the AAAV program 





to adopt a different strategy during SDD. 





First, because the RRB relied on the senior 
program leadership, its frequency of meeting became 


difficult during especially busy time periods of PDRR. The 





PDRR risk management process depended on the RRB to provide 


guidance and decision-making to function efficiently and 





consistently. The RRB, though a good idea and effective 
when executed, lacked the sustainability characteristic 


needed in SDD. 


Secondly, some personnel from DRPM AAA and GDAS 





believed the RRB became a forum that did not always embrace 
risks as opportunities or foster an environment of open, 


non-attribution discussion. One GDAS employee said that 





the RRB became a “moot court.” The result was a 





disincentive for IPTs or stakeholders to identify and 





introduce risks into the risk management process. The DRPM 
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AAA has responded by establishing the PMT II meetings, 


which serve as the Risk Coalition Team (RCT). 





The RCT is “a team of Government personnel 
chartered as owners of the risk management process. Its 


primary purpose is to facilitat the creation and 





continuous operation of the risk management process.” 
(Draft DRPM AAA SDD Risk Management Plan, November 2002) 
The SDD RMC (Program Integration Branch Head) chairs the 


RCT. The RCT includes members from the following DRPM AAA 








directorates: 
e PP&l 
° Systems Engineering 
° Logistics 
° Testing 
ad Cost 
° Engineering representatives 
e DCMA representatives (Draft DRPM AAA SDD Risk 


Management Plan, November 2002) 


The biggest difference between the RRB used in 





PDRR and the RCT used in SDD is the organization’s 





membership. The RRB consisted mainly of Division Directors 
whereas the RCT is comprised of the next lower level, th 
Department Heads or PMT II members. The benefit in the 











revised structure is that the RCT performs the majority of 
detailed risk analysis so they can then provide 
recommendations and courses of action to the PMT. The RCT 
can filter and solve many problems before coming to the 
attention of the PMT allowing the PMT to focus on higher 
level, Government-specific risk areas. The RCT meets bi- 


weekly as part of the PMT II meetings. 
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Another risk management technique the DRPM AAA 
used during PDRR- was inviting an Independent Risk 
Assessment Team (IRAT). The purpose of the IRAT in PDRR 


was to evaluat th effectiveness of the program’s risk 





management process and provide recommendations for SDD. 

c. Periodic Risk Assessments 

During PDRR, the AAAV Team conducted periodic 
internal and external risk assessments of the program’s 


risk management process. Prior to Milestone II, the AAAV 





Team requested an independent risk assessment to evaluate 


the status of the program’s technical risks and the risk 





management process prior to entry into the System 





Development and Demonstration (SDD) Phase. 


EG&G, DRPM AAA’ s risk management support 





contractor, nominated a “three-member team of experienced 





engineers” to conduct the assessment. The Independent Risk 





Assessment Team (IRAT) was made up of EG&G Technical 








Services representatives and chaired by a representative 


from the Tllinois Institute of Technology Research 





Institute who were also co-authors of the NAVSO Guide “Top 


Eleven Ways to Manage Technical Risks.” (IRAT Report, 





September 2000) 


The DRPM AAA tasked the IRAT with conducting an 
impartial assessment of the AAAV risk management program. 


The AAAV DRPM planned to integrate the IRAT’s findings into 








part of the Milestone II Defense Acquisition Board (DAB) 





documentation. (AAAV Independent Risk Assessment Brief, 


March 2000) 


The risk areas of interest were product and 


technical process risks. (IRAT Report, 2000) The AAAV Team 
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intended to use the IRAT findings to prepare for the 


upcoming Milestone II decision, identify technical and 





process risks associated with entering SDD and evaluate 


risk-handling procedures during PDRR. The DRPM provided 








IPT briefings and disclosed extensive technical and program 





documentation to facilitate the assessment. 


The IRAT’s methodology was as follows: 


e Review related program documents, such as AAAV 

Risk Management Plan, SEMP, Management Plan, ORD, 
TEMP and both the Program Definition and Risk 
Reduction (PDRR) and EMD SOWs 











DRPM AAA IPT leads briefed the IRAT regarding 
implementation of risk management, issues/risks, 
and status 








e Follow-up with interviews, discussions and data 
gathering in selected areas 


° Prepare the IRAT report and out brief results to 
the DRPM AAA (IRAT, September 2000) 


The objective of the assessment was to provide an 





objective overview of the program’s risk management process 


and identify risk areas for entry into SDD. 


The DRPM AAA found the IRAT to be extremely 
useful. The IRAT’s objectivity on the risk management 


process provided invaluable feedback that allowed the AAAV 





program to shape its future SDD processes. The DRPM AAA 





intends to conduct periodic IRATs during SDD and in 


preparation for the LRIP decision. 


The risk management process discussed in the 
preceding section analyzed and introduced several of AAAV’s 


risk management techniques used during PDRR and SDD. The 





next analyzes one aspect of the AAAV program’s use of the 
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contracting process to reduce programmatic risks and 
transfer risk from the Government to the contractor. 


3. AAAV Risk Management Through the Contracting 
Process 


Following the Milestone I decision, the Government 


awarded GDAS a Cost Plus Award Fee (CPAF) type contract for 


PDRR. Part of the second period award fee criteria was 
tied to GDAS’ risk management process. Using contract 
award fees to provide contractors with incentives to 





proactively manage risk is an effective risk mitigation 





activity. 


GDAS uses an award fee sharing system among its 




















employees. Each GDAS employee earns a portion of the 
contract award fee. A graduated scal determines th 
various amounts. Employees earn percentages of award fees 
according to their position within GDAS. Award fee 


sharing, or profit sharing, in organizations motivates good 





behavior and provides incentive for all contractor 


employees to perform. 


As a result of the award fee assessment, GDAS placed 
higher priority on risk management, from both a capability 


as well as training aspects. It was at this time that GDAS 





developed the VDD risk management applications and 


formalized the process used during PDRR. 





The lesson learned was that contractually obligating 





and providing incentives for the Prime contractor to 





spearhead risk management efforts transferred some 
accountability for risk from the Government to industry. 
The Government was then prepared to evaluate the Prime 


Contractor’s performance and reward them accordingly 
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through contract period award fees. The current SDD 
contract type is also a CPAF with award fee criteria tied 
to GDAS’ risk management process. This also allows the 


Government to supervise GDAS’ technical and system-level 





risk management while concurrently implementing a 








Government-specific risk plan. 





GDAS’ award fee sharing system sustains employ buy 


in by financially rewarding its people for good 





performance. The DRPM AAA’s contracting strategy is a good 
example of risk transference in DoD systems acquisitions. 


Once implemented, the AAAV risk management process relied 





heavily on Integrated Product Team (IPT) interaction and 


the Integrated Product and Process Development (IPPD) 











process. The next section analyzes the AAAV program’s 
unique co-location and IPPD organization. 

4. AAAV PDRR Co-Location and IPPD Process 

The AAAV program’s co-location at the Woodbridge, VA 


facility fosters continual communication and interaction 





between Government and GDAS personnel. This author 


interviewed both DRPM AAA and GDAS individuals concerning 














co-location. Both groups were in agreement that co- 
location allows for a great degree of real-time 
communication, which can reduce design risks, especially in 





the PDRR phase. Synthesis is a vital step in the systems 





engineering process and communication is imperative to 





effective synthesis, as decisions are being made with the 





full understanding of the Government through its 





counterparts on the IPTs. In addition to the communication 


benefits, both DRPM and GDAS personnel agreed that co- 











location helps each group understand the other’s culture 





and therefore develop more effective working relationships. 
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In addition to the office space co-location, the vehicle 
assembly and production area located at the joint facility 


was of great value. 


Co-location at the point of vehicle assembly allowed 
nearly seamless system integration activities. This 
facility reduced the risk of engineering and schedule 


problems by allowing GDAS and DRPM decision-makers to apply 





timely resources to friction points and fully understand 





the risks being mitigated. 


During a visit to the Worth Avenue Technology Annex in 


Virginia, this author saw active duty Marines, 





representative of the intended end-users, providing input 


during design and assembly of SDD prototypes. This type of 





interaction greatly reduces the risk of systems. not 
adequately accounting for logistics and  supportability 


considerations during the design stages. Had GDAS and DRPM 





AAA not been co-located, it is doubtful that user 
representatives would have as much opportunity to provide 


input and feedback in the design stages. The alternative 





can result in time-consuming and costly system changes 








rior to fielding or once a system is fielded. 











p 
Additionally, funding usually necessary for travel to bring 
I 


PTs or program leaders together is avoided by co-locating. 


Co-location is synonymous with the principles of 
Integrated Product and Process Development (IPPD). 
Continual cross-functional area communication is a large 


part of the IPT process. Co-location allows the AAAV 








program to operate in an IPPD environment. 
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a. Integrated Product and Process Development 
(IPPD) and the AAAV Program 


The AAAV Team’s co-location facilitates the IPPD 
process and allows IPTs to work and interact on a frequent 
basis during SDD. The AAAV co-location as an enabler of 


the IPPD process “provides the foundation and communication 








conduits for the IPTs to maximize th effectiveness of 
every member of the organization.” (Pollution Prevention: 
AAAV, October 2002) The Government and GDAS co-location 
facilitates IPT integration and interaction between the 
Government and company engineers, Jlogisticians, product 


managers, and other functions. The program’s emphasis on 





cross-functional coordination and efforts has and will 





result in significant schedule and cost risk mitigation and 











avoidance. In particular, the AAAV’s dedication to 





reducing and, in som cases, liminating environmental 





safety hazards (ESH) in the system’s production will reap 





noteworthy Total Ownership Cost (TOC) and Lifecycle Cost 








(LCC) avoidances. Each AAAV IPT has an ESH representative. 





The following section is a case study of the benefits of 
IPT co-location and IPPD interaction. 


b. Environmental Safety and Health (ESH) and 
the IPPD Process 


Environmental Safety and Health (ESH) 


considerations may constitute significant programmatic 





risks. Such risks include safety of use, environmental 
impact of system use and exorbitant system disposal costs. 


In response to environmental risk considerations, the AAAV 





DRPM established an ESH Working Group (ESH WG). The ESH WG 











was tasked to “identify, evaluate, track and assist with 





mitigation of ESH hazards.” (Pollution Prevention: AAAV, 


October 2002) The ESH WG’s membership included both 
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Government and GDAS personnel who possess experience with 





ESH issues and environmental considerations. AAAV IPTs all 








have ESH representatives. The representatives ensure that 
ESH considerations and impact are part of all development 


and production decisions in the IPPD process. 


The ESH WG created the “first ever Risk Reduction 





Process, mbedded in the VDD, to identify, track, and 





eliminate ESH hazards.” (Pollution Prevention: AAAV, 





October 2002) The ESH WG identified and assigned “over 











five hundred ESH risk hazards” and “developed the ESH 





Database that provides the communication and tracking link 
for these hazards, the identification of the lead IPT for 


mitigation action, and tracking of the risk hazard 








resolution/acceptance.” (Pollution Prevention: AAAV, 


October 2002) 











The DRPM AAA developed an ESH Awareness Session 
for all members in AAAV IPTs. (Pollution Prevention: AAAV, 
October 2002) The purpose of the ESH awareness session was 


to train and educate AAAV Team members on the potential 





risks associated with ESH. 


The DRPM AAA drafted the system’s performance 
specifications to include a ban on all Class I and II Ozone 


Depleting Substances (ODS) in the design and manufacture of 





the AAAV. Additionally, the Government has contractually 


obligated both Prime and Subcontractors to eliminate the 





use of cadmium, lead, chromium and other environmentally 
hazardous materials in the production of the AAAV. The 


deletion of these environmentally harmful substances will 





reduce the risks of negative environmental impacts and high 





disposal costs. 
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Over the life of the AAAV program, the AAAV 
anticipates cost avoidance of $379.9 million in production 
and $238.9 million in Operations and Support (O&S) costs. 


(Pollution Prevention: AAAV, October 2002) The AAAV ESH 











initiatives have played a significant role in the projected 


cost avoidances through the application of the ESH Working 





Group (ESH-WG) and the Virtual Design Database. 


The U.S. Army’s Center for Health Promotion and 





Preventive Medicine (CHPPM) and the U.S. Navy’s Explosive 











Safety Review Board (WSESRB) stated that the AAAV’s “ESH -WG 
Risk Reduction Process is the best program of its type that 


they have encountered in DoD.” (Pollution Prevention: AAAV, 





October 2002) The ESH-WG IPT representation ensures that 








ESH considerations are input throughout every design and 


manufacturing decision made for the AAAV. Through the 





system decomposition and iterative design process, ESH 





input in the IPT process enhances the likelihood that the 
system is built correctly the first time. This eliminates 


costly design or manufacturing process changes after the 











system is fielded or prior to demilitarization. The ESH -WG 


and ESH IPT representation are examples of the benefits of 





the IPPD process in weapon system developments. ESH risks 





were incorporated into the program’s risk management 


process during PDRR in the same manner as all other risks 





managed by IPTs. 


The relational VDD, discussed in preceding 





sections, allowed th ntire AAAV IPT structure to have 





visibility on ESH risk identification, tracking, resolution 





and documentation issues. The VDD has resulted in 


effective horizontal, vertical and cross communications 
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concerning ESH risk handling. Because risks in VDD 





corresponded to WBS elements, the cause and effect 





relationships of ESH risks were easy to identify and 


assess. 


The AAAV program has taken proactive measures to 


reduce the system’s environmental impact. These measures 





will lead to cost avoidance over the life of the program as 














well as preserving the environment. The continual ESH 





= 


assessment during the IPT process may have been less 
effective or ineffective had the AAAV program not been co- 


located during PDRR. The program’s co-location and 








commitment to IPPD enabled ESH considerations to be part of 


design trade offs and CAIV analysis. 


The AAAV performance specifications negate the 


use of several environmentally hazardous materials in the 





system’s production. Offerors have had to develop and 
incorporate new, environmentally safe materials for 
integration into the AAAV. For example, cadmium was 


eliminated because of the presence of cyanide and other 
hazardous materials (HAZMAT) in the plating process. 


(Pollution Prevention: AAAV, October 2002) 


Furthermore, the AAAV is testing a developmental, 





water-based Chemical Agent Resistant Coating (CARC) paint 


that will reduce Hazardous Air Pollutants (HAP) during 





manufacturing and repair of the system. The use of a 





water-based CARC paint is expected to save $2.8 million 
over the life of the program. (Pollution Prevention: AAAV, 
October 2002) The new CARC paint may represent enormous 
savings throughout DoD for systems that require CARC. Such 


developmental innovations may reduce future, developmental 
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system costs and reduce the risk of acquisition funds being 


diverted from procurement to support unanticipated O&S 





COSts. 


Working with the U.S. Environmental Protection 





Agency (USEPA), the AAAV program has also eliminated the 
use of all Class I and II Ozone Depleting Substances (ODS) 





and the majority of the USEPA’s top seventeen hazardous 


materials. Savings as a result of the elimination of these 





hazards is expected to be in the tens of millions of 





dollars with even greater potential with DoD-wide adoption. 





Reduction of HAZMAT initiatives during the AAAV’s 


design and manufacture directly impacts Total Ownership 


Costs (TOC). The AAAV’s avoidance of ODS and other HAZMAT 





will allow the system to enter its disposal phase without 





Significant commitment of additional resources to make the 


disposal environmentally friendly. The AAAV’sS insistence 





that the Prime and Subcontractors use environmentally 


improved materials will result in savings in future systems 

















LCC and Research, Development, Test and Evaluation (RDT&E) 


activities. 


ESH risk mitigation is just one example of the 





AAAV IPPD process. Although other DoD programs that are 





not co-located effectively operate in an IPPD environment, 
the AAAV'sS co-location encourages continual, cross-— 


functional area communication and constant interaction 





between GDAS and the DRPM AAA. In order to determine th 








effectiveness and supportability of ESH initiatives, the 
AAAV team needs to thoroughly test the system. The next 
section analyzes the AAAV PDRR test and evaluation plan. 
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5. AAAV PDRR Test and Evaluation 





The aggressiv developmental testing (DT) and early 





operational assessment (EOA) performed during PDRR will 
benefit the AAAV program during SDD and _ subsequent 


production and fielding. The results of the DT events and 





the EOA in PDRR allowed the AAAV team to identify what 





areas of the system required further developmental 
attention and testing. The PDRR test results helped 
determine th SDD test plan. On using test as a risk 





management tool, the DAU writes: 


Fixes instituted during early work efforts 
(Systems Integration) in the System Development 
and Demonstration (SDD) Phase cost significantly 
less than those required in later System 
Demonstration after the critical design review 
when most design decisions have been made.” (Test 
and Evaluation Management Guide, November 2001) 














(Tv ct 











Early testing can reduce the risk of run-away system 








delivery costs and expensive design changes late in the 





acquisition process. By conducting an EOA during a 


Combined Arms Exercise (CAX) in 29 Palms, California, the 








AAAV program combined developmental and operational test 
(OT) activities. DT under operational conditions, similar 


to those specified in the Operational Requirements Document 





(ORD) , can reduce time and costs through concurrent 








testing. When conducted too late in a program’s schedule, 
combined DT and OT events can result in OT failures and can 


impact milestone decisions. Conducted early, the program 





can identify risk areas and develop mitigation and test 





plans to fix and validate deficiencies. The DoD 5000.2-R 


encouraged combined DT and OT testing: 
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A combined developmental test and evaluation 
(DT&E) and operational test and evaluation (OT&E) 
approach should be considered when there are time 
and cost savings. The combined approach must not 


compromise either DT or OT objectives. (DoD 
5000.2-R, April 2002) 














Data obtained from combined DT and OT events can be 
collected and potentially used in lieu of follow-on test 
events. Coordination with service test agencies and the 


Director of Operational Test and Evaluation (DOT&E) is 





important for obtaining and using test data. 


Another lesson learned from the AAAV gunnery EOA was 


that crew training and experience on the weapon system is 





critical to successful test execution. The PMO is applying 





this lesson learned to the test crew-training plan during 


SDD. 


In addition, the program office conducted xtensiv 





testing on the MK46 weapons station to include a gunnery 








EOA. The MK46 lethality tests met or exceeded all ORD 
requirements during PDRR. In the course of testing, the 
AAAV program identified areas that will require additional 








study and follow-on testing in SDD such as ventilation and 








gunner training. This knowledg helped th DRPM AAA 
develop the SDD test plan and concentrate on specific 


system components prior to additional OT. 


Other than the expected system performance-oriented 


test events, the AAAV program conducted extensive lif 





cycle support testing during PDRR. 


Life cycle support testing included logistics 


demonstrations, maintainability demonstrations, mean time 





to repair (MTTR) demonstrations and human factors 
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engineering testing. The significance of the AAAV’s PDRR 
test plan was its emphasis on life cycle support. Life 


cycle support considerations will drive the AAAV’s life 





cycle costs (LCC). Some estimates report that nearly 75 - 
80% of a system’s overall cost is assumed during Operations 


and Support (O&S). 








Test events designed to validate ORD-specified MTTR or 


operational availability parameters can reduce LCC and poor 








operational availability. This can be achieved by 


including user juries during logistics demonstrations and 








MTTR demonstrations. User juries can identify system re- 
design recommendations to make the system more 
maintainable. The system’s supportability will drive its 
operational availability. Decreased repair requirements 
and cycle times will reduce maintainability costs and time. 











The result will be a supportable and available system for 


the intended user. 





Logistics can be a primary system cost driver during 
O&S. Thoroughly testing a system during PDRR and SDD for 
supportability, reliability and maintainability can greatly 


reduced O&S costs and increases operational availability by 





identifying supportability risks early on in the program’s 
schedule. Reducing the risk of unanticipated O&S costs can 
protect resources intended for pre-planned program 


improvements (P3I) and new developmental systems. 


The SDD test plan will focus on verifying design 








improvements identified in PDRR by testing an expected nine 





SDD prototypes. (Developmental Testing Brief, November 





2002) The emphasis will be on addressing design changes 
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identified in PDRR and conducting OT activities in order to 
meet Low Rate Initial Production (LRIP) entrance criteria. 
E. SUMMARY 

This chapter analyzed the data presented in Chapter IV 
of this thesis. This chapter discussed the author’s 


purpose of research and research methodology. The primary 





purposes were to analyze and discuss the AAAV PDRR risk 





management strategy and connect the lessons learned in PDRR 


to the SDD risk management techniques. 


This chapter discussed AAAV PDRR information 
technology tools: VDD, VINTEGRA and LCIS. The lesson 





learned from PDRR was that information technology tools 
could be force multipliers in managing risk. VDD served as 
a central repository, or risk database. VDD risk 


management applications facilitated the formal risk 





management process. VDD’s automatic notification system 
enabled communication across program functional lines. 
Shortfalls in VDD have led to the development of LCIS 
during SDD. LCIS aims to create a risk management tracking 
application that emphasizes trend analyses, reporting and 


program-wide risk management. 





VINTEGRA continues to reduce system production risks 


by identifying integration and assembly and _ production 








refinement requirements. Additionally, VINTEGRA will 


reduce cost risks in the development of IETMs. The program 











office expects IETMs to reduce maintenance delay times thus 


improving operational availability and logistics strains 





during O&S. 


The AAAV PDRR risk management process was effective. 


However, the program office learned valuable lessons from 
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its process and is applying lessons learned to the process 





used in SDD. Namely, the DRPM AAA and GDAS instituted the 








PP&l directorate. One of the functions within PP&I is the 
overall risk management process. The program implemented 
the PMT II bi-weekly meetings. Risk management is a 


statutory PMT II agenda item. Through the PMT II, the RCT 





will manage program risks and develop courses of action to 





present to the program leadership for decision support. 
During PDRR, the Government incentivized risk 
management through the use of an award fee. The CPAF 


contract type provided financial incentives 





execute sound risk management practices. 


awarded GDAS a CPAF type contract for § 


for GDAS to 
[The Government 


DD with risk 





management tied to award fee criteria, as wel 


Flee Including 


financial incentives in contract award fee criteria is an 








ffective technique to transfer risk from the 








Government to 


a Prime contractor. Both entities gain from effective risk 


handling as a result. 


The DRPM AAA has co-located with its Prime contractor, 


GDAS, in Northern, Virginia. The AAAV program co-location 


has facilitated continual communication an 


between Government and Industry personnel. 





d interaction 





The close 


working relationship is consistent with IPPD principles. 





The AAAV program’s IPPD process has benefited from the 


communication advantages provided by co-locati 





Finally, the AAAV PDRR~- test plan 


on. 


included OT 


activities combined with DT. The aggressive test plan 


allowed the program to identify areas that 





will require 


greater attention during SDD. The results of the PDRR DT, 


OT and EOA have helped shape the SDD test plan. The SDD 
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test strategy is to fix design deficiencies discovered 


during PDRR and prepare the system to meet LRIP entrance 





criteria. 


The next chapter concludes this thesis. The purpose 
of the conclusion is to discuss how the AAAV SDD risk 


management strategy reflects the lessons learned from the 





PDRR risk management approach. Additionally, the chapter 





discusses which elements of risk management practices the 
AAAV program is benefiting from most and provides 
recommendations for managing risk in developmental weapon 


systems. 
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VI. CONCLUSION 


A. INTRODUCTION 


This chapter provides conclusions and recommendations 


drawn from the analysis of the Advanced Amphibious Assault 





Vehicle (AAAV) Program Definition and Risk Reduction (PDRR) 





and System Development and Demonstration (SDD) risk 








management strategies. The benefit of this research is to 
illustrate risk management techniques used in Department of 
Defense (DoD) weapon system procurement and development 
through a study of the AAAV’s transition from PDRR to SDD. 
B. CONCLUSIONS AND RECOMMENDATIONS 

This section discusses conclusions regarding risk 
management procedures used by the AAAV program during SDD. 
Based on the conclusions, this chapter offers 


recommendations for managing risk in DoD developmental 





programs. 
1. AAAV Information Technology Tools 
a. Conclusions 
Weapon system programs that use Information 
Technology (IT) tools or applications to complement risk 
management processes can effectively manage many of the 
risk areas discussed in this thesis. The AAAV program used 


the Virtual Design Database (VDD) during PDRR to augment 





the formal risk management process. 


VDD enabled the program management office (PMO) 
to identify, categorize, communicate and file risks through 
a relational database available on the program’s Intranet. 
Acknowledging the need to expand VDD’s capabilities to 
manage risks during SDD, the AAAV PMO is developing Life 


Cycle Information System (LCIS). LCIS expands on VDD by 
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incorporating applications that assist in tracking risk 
trends and conducting variance analysis to assess 
mitigation efforts: a capability the PMO recognizes as 


important in managing risks in SDD. LCIS, a web based 





application, will improve the program’s communications by 








making the application available to sub contractors and 
potentially to remote test locations, as well. It is being 


developed to support the anticipated future needs of the 








AAAV as a Program Management Office of Life Cycle Support 
(PMOLCS) . 


The benefits that IT applications provide PMOs 
are numerous. LCIS will expand on VDD’s ability to 
communicate emerging risks, track risk mitigation 
activities and conduct risk analyses to support program 
level risk management efforts. 

b. Recommendations 

Based on the conclusions discussed above, this 
thesis offers recommendations for managing risk in weapon 


system programs through the use of IT applications: 


° Develop and employ electronic resources to 
facilitate and complement the program’s formal 
risk management methodology 











° Make this IT resource available to all program 
stakeholders: Government, Support Contractors, 
Prime and Sub Contractors 











e Ensure all users are properly trained 

e Keep the application simple 

° The application should support the program’s 
specific goals or efforts in an acquisition phase 





° Anticipat desired expansion of the tool’s 
capabilities to satisfy program requirements in 
later program phases 
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° When practical, employ a dedicated program Chief 


Information Officer (CIO) to oversee IT 
initiatives 
2. The Joint Government-—General Dynamics Risk 


Management and Resolution Process 
a. Conclusions 
A weapon system’s risk management process should 


be simple and sustainable. The success of a process will 





depend in large part upon the degree to which its 
implementation does not detract from concurrent program 


demands. Risk management is a continuous part of the 





system development and acquisition process. All activities 


in the process should add measurable value to the program’s 








development and production. Risk mitigation activities 
need to be tied to metrics to evaluate progress and 


efficacy of the efforts. 


The formal risk management process should focus 
on what is important to the program: meeting user 
requirements given time and resource constraints. The 


process should involve senior leadership participation 





while maintaining an environment that encourages the airing 
and resolution of risks. It should reward initiative and 


acknowledge the value of responsible risk acceptance while 





insisting on accountability and ownership of risk and its 
mitigation. 

b. Recommendations 

The analysis and conclusions of the AAAV’s risk 
management and resolution process drive the following 


recommendations: 
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The risk management process should be simple 


A program management office should consider using 
a risk primer to familiarize and train personnel 
in the risk management process 


A formal risk management process should be 
sustainable given program office workforce 
strength and anticipated demands 


Risk management activities should add measurable 
value to the program 


The establishment of a program directorate or 
division to oversee risk management can help to 
coordinate efforts, track risk trends and liaise 
with leadership 


Metrics should be developed and employed to 
assess the success or failure of mitigation 
efforts 


Program offices should consider periodic, 
external risk management assessments 


Risk Management Through the Contracting Process 
a. Conclusions 


Contract incentives tied to a Prime contractor’s 


risk management process are an effective tool to transfer 





risk from the Government to its industry counterpart. The 





Government program office ultimately assumes responsibility 


for the 


However, 


success or failure of a system’s risk handling. 


tying financial incentives for the contractor to 








develop and implement effective risk management processes 





can result in improved contractor performance. GDAS’ award 
fee, profit sharing structure resulted in increased 
mploy buy-in to the AAAV risk management process during 
PDRR. As a result, the Government continues to provide 





contractual award fee incentives for GDAS to execute a 





proactive risk management process in SDD. 
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b. Recommendations 





As a result of the success the DRPM AAA has had 


in tying risk management to GDAS’ contract award fee plan, 





this thesis offers the following recommendations concerning 


risk management through the contracting process: 


° Transferring risk from the Government to 
contractors through financial incentives can be 
an effective method to achieve desired results or 
levels of effort 














° Profit-—sharing organizational structures can 
incentivize good performance and employee buy-in 








4. Government and Prime Contractor Co-Location 
a. Conclusions 
The Marine Corps/GDAS co-location on the AAAV 
program has created an environment of continual 
communication between Government and Prime contractor 
personnel across traditional program lines. The ease of 
communication and problem resolution decreases 


administrative delay times and miscommunications between 





Government/contractor counterparts. Co-location at the 


point of vehicle design, testing, assembly and prototype 





production nhanc th IPPD process by providing an 








environment for teams to truly integrate. 


The benefits of AAAV’s co-location are evident in 


the proactive management of risks such as those in 





Environmental Safety and Health (ESH). Co-location allows 
Government and Industry personnel to identify and 


appreciate different organizational cultures. 





Understanding these differences nables both the Marine 


Corps AAAV team and GDAS to work towards establishing the 








most productive work environment for the benefit of the 


program. 
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b. Recommendations 


Recommendations concerning Government PMO and 
Prime contractor co-location are as follows: 


e Co-location facilitates communication and _ the 
IPPD process 








° Co-located PMOs can save time and money in a 
system’s developmental stages 








° User representatives involved on a regular basis 
in a system’s design for suitability can prevent 
costly and avoidable changes 


5. Test and Evaluation 
a. Conclusions 


The AAAV's test program in PDRR included 


combining DT and OT test events in a challenging 








operational environment. The lessons learned from the test 
results allowed the program office to know what its 
strengths and deficiencies were ntering SDD. This 
knowledge shaped the SDD test plan to prepare the system to 














meet LRIP entrance criteria. Testing the system 
aggressively and early in the test plan reduced the risk of 


being forced to combine risky DT and OT events prior to a 





Milestone decision because of schedule compression. 


Including user juries and Fleet Marine Force (FMF) Marines 








in the EOA provided the program with relevant user feedback 
early in the system’s design stages when changes were 
possible and less costly. The AAAV’s emphasis on life 


cycle support testing in PDRR represents the Marine Corps’ 





goal of reducing total ownership cost of the AAAV. 
b. Recommendations 
Based on the AAAV’s test history, this thesis 


offers the following recommendations: 
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° Conduct logistics and life cycle support tests 
with user juries early in the system’s 
development to avoid costly changes and reduce 
the risk of fielding an unreliable or difficult 
to maintain system 





° Combining DT and OT test events during PDRR 
allows a program to refine its SDD test plan to 
successfully meet LRIP entrance criteria 





° Include user juries in DT and OT test events 
whenever feasibl to solicit feedback early in 
the system’s development 











° Ensure all personnel involved in test events are 
thoroughly trained and have sufficient experienc 
to be able to execute required activities at an 
acceptable level of performance. 

















This chapter concludes the thesis by addressing 
what conclusions and recommendations can be made from the 


analysis of data presented in previous chapters. The 





following section provides suggested areas of further 
research in risk management and in the AAAV program. 
c. AREAS OF FURTHER RESEARCH 

The following areas of further research are suggested 
to expand upon this analysis of current DoD risk management 
practices in the development and procurement of complex 


weapon systems: 


° What impact do the revised DoD 5000 £Series 
acquisition guidelines have on DoD developmental 
weapon system risk management practices? 


° How do the revised DoD 5000 Series acquisition 
guidelines impact the AAAV program prior to and 
Following Milestone C? 


e What conclusions and recommendations can be made 
from an analysis of the AAAV program’s SDD risk 
management strategy? 


e What conclusions and recommendations can be drawn 


From an analysis of co-located and detached 
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program management offices regarding the impact 
of co-location on the IPPD process? 


What risk management techniques are being used to 
manage software intensive programs? 








What metrics can be used to evaluate software 
development and are they effective? 
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APPENDIX. LIST OF ACRONYMS 


Advanced Amphibious Assault Vehicle 


Amphibious Assault Vehicle 


Analysis of Alternatives 





Acquisition Program Baseline 








Advanced Technology Demonstration 
Computer Aided Design 


Cost as an Independent Variable 





Center for Naval Analyses 





Chemical Agent Resistant Coating 
Combined Arms Exercise 


Center for Health Promotion and 
Medicine 


Chief Information Officer 
Cost Plus Award Fee 
Computer Systems Corporation 


Defense Acquisition Board 





Defense Acquisition Deskbook 





Defense Acquisition University 





Department of Defense 


Direct Reporting Program Manager 


Amphibious Assault 





Developmental Test and Evaluation 
Engineering Change Proposal 

Early Operational Assessment 
Executive Program Managers Course 


Environmental Safety and Health 





Earned Value 


127 


Direct Reporting Program Manager 


Preventive 


Engineering and Manufacturing Development 


Electronic Problem Resolution System 


Advanced 























Earned Value Management 








Earned Value Management System 
Fleet Marine Force 
Government Accounting Office 


General Dynamics Amphibious Systems 





Goal, Question, Metric [paradigm] 


Guidelines for Successful Acquisition of 
Software-Intensive Systems 








Hazardous Air Pollutants 





Interactive Electronic Technical Manual 





Integrating Integrated Product Team 


Integration and Assembly 





Initial Operational Capability 


Integrated Product and Process Development 








Integrated Product Team 
Independent Risk Assessment Team 


Information Technology 








Integrated Test Program 

Joint Total Asset Visibility 

Key Performance Parameter 

Landing Craft Air Cushioned 

Lifecycle Cost 

Life Cycle Information System 

Low Rate Initial Production 

Marine Corps Combat Development Command 
Milestone Decision Authority 

Mission Needs Statement 

Modeling and Simulation 

Milestone 


National Environmental Protection Act 








National Military Strategy 

















National Security Strategy 
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Operational Maneuver From the Sea 


Ozone Depleting Substance 


Operational Requirements Document 


Operations and Support 





Operational Test and Evaluation 


Program 


Program 
Program 


Program 
Support 


Program 





Program 


Request 





Personal Computer 





Definition and Risk Reduction 


Pre-planned Program Improvement 


Manager 





Management Office 


Management Office of Life Cycle 


Management Team 


Planning and Integration 


Risk Coalition Team 





Research, Development, Test and Evaluation 


for Proposal 


Risk Management 
Risk Management Coordinator 
Risk Management Plan 


Risk Resolution Board 





System Development and Demonstration 








Software Engineering Institute 








Statement of Objectives 





Statement of Work 


Source Selection Process 





a 





United 
Agency 





echnical Data Package 


est and Evaluation 





est and Evaluation Master Plan 


Total Ownership Cost 





Technical Performance Measurement 


States Environmental Protection 
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VDD 
VINT! 





eal 
Q 
3 


WATA 
WBS 

WG 
WSESRB 





Virtual Design Database 





Virtual Integration and Assembly 





Worth Avenue Technology Annex 
Work Breakdown Structure 


Working Group 





Weapon System Explosive Safety Review Board 
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